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ABSTRACT 

This paper describes a four-step approach to 
constructing a diagnostic test that provides precise but practical 
information on students' problems and needs for additional 
instruction or remediation. The approach is based on analyzing the 
structure of the domain to determine which skills within the domain 
need to be assessed to diagnose students' problems. The four steps 
include: (1) identifying the factors that describe the curricular 
domain, (2) constructing a test with items representing all possible 
combinations of content and cognitive factors, (3) determining which 
factors and interactions among them produced variations in students' 
scores using generalizability theory, and (4) determining the minimum 
number of items needed to obtain a generalizable measure of each 
skill in the diagnostic profile. This paper contains three studies: 
"Optimizing the Diagnostic Power of Tests: An Illustration from 
Language Arts, 11 by Moreen Webb, Joan Herman, and Beverly Cabello; 
"Diagnosing Student Errors: An Example from Science," by Steve Shaha; 
and, "Task Structure Design: Beyond Linkage," by Eva Baker and Joan 
Herman. Two of these studies explicity address problems related to 
the design of diagnostic tests; the third considers a conceptual 
model for integrating testing and instruction. (PN) 
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PREFACE 




CSE's Methodology Project during FY1983 had two primary .emphasis: 
diagnostic testing and comprehensive evaluation systems for local 
school improvement. Within the former area, CSE conducted a series of 
research activities to investigate the feasibility and potential of 
diagnostic -testing for classroom u se. Preparatory to the deve lopment 
of a revised microcomputer-aided diagnostic testing system, the 
research addressed issues in test design and analysis and in the 
potential and applications of alternative models of diagnosis. The 
results of each of these inquiries is reported separately. 

This document reports on CSE's research efforts in the area of 
test design. Of the three studies included here, two explicitly 
address problems related to the design of diagnostic tests; the third 
considers a conceptual model for integrating testing and instruction. 

The first paper, "Optimizing the Diagnostic Power of Tests: An 
Illustration from Language Arts," investigates strategies for 
improving the diagnostic power of test items so that they provide more 
precise but practical information on students' problems and needs. 
Based on a domain-referenced approach, the study examines factors 
which may be diagnostically useful in profiling students' performance 
and explores methods for analyzing and structuring diagnostic tests. 

The second paper, "Diagnosing Student Errors: Ah Example from 
Science," investigates the effects of cognitive level on student test 
performance and examines the utility of particular error types in 
characterizing student strengths and weaknesses. 

' ' ; 3 

ERIC 



The final paper, "Task Structure Design: Beyond Linkage" 
presents a conceptual model for designing testing to maximize the 
integration of testing and instruction and to maximize the utility 
test results for instructional decisionmaking. 
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Introduction 

Assessment has an integral role to play in the Improvement of 
instructional practice. Mastery learning strategies (Bloom, 1976; 
Block, 1971), systematic instruction (Popham and Baker, 5 1976), 
individualized instruction (Glaser, 1970; Klausmeier 1976) ; clinical 
teaching (Hunter, 1983) and effective schooling (Edmonds, 
1981) all point to the importance of assessment in diagnosing 
students' strengths and weaknesses, in monitoring their progress 
through the curriculum, in providing instruction that 1s tailored to 
instructional needs and goals and thus 1n enhancing student 
achievement. The underlying theory derives from a systems view of 
education and suggests that 1f teachers are to maximize tn^ir 
students 1 learning, they need to: plan Instruction on the basis of 
the needs of Individual or groups of students; to monitor their M 
progress; to determine whether remediation 1s required; and to 
evaluate outcomes to assess the success of Instruction, as well as 
needs for modification and students 1 readiness for succeeding work. 

Diagnosis and prescription is thus a recurring concern throughout 
the Instructional process and is central to its success. Yet despite 
its Importance, the assessment tools teachers have available for such 
a process are really quite limited. While so-called diagnostic tests 
do exist, the level of Information they provide is less than optimal. 
A typical diagnostic test in reading, for example, may characterize 
student* needs by providing a total score and subscores for individuals 
and groups In such areas as vocabulary, literal comprehension, 
inferential comprehension, etc, but such scores offer teachers little 
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guidance regarding the nature of any reading problems or their 
causes. It is left to the teacher to pinpoint why students perform as 
they do and to prescribe instruction accordingly. In contrast to this 
global approach, more recent research has taken a molecular view of 
the diagnostic problem. Tatsuoka and associates (1980) for example, 
have completed extensive work in diagnosing student performance in a 
very narrow mathematics domain (the subtraction of two digit signed 
numbers) and have identified the specific misconceptions and . 
difficulties which students manifest in this area, e.g., six specific 
error types related to determining the sign of answers. While the 
advent of classroom computer technology may make such advances more 
useable in future classroom practice, these findings .provide a 
of detail beyond the grasp of today's teachers: a teacher cannot 
track a classroom of students across so many error dimensions for all ^ 
curriculum areas, nor feasibly tailor instruction at this level of 

specificity. - 

The current study seeks an intermediate level for constructing 
and analyzing diagnostic tests for classroom use. It Investigates 
strategies for Improving the power of diagnostic Instruments so that 
they provide more precise but practical Information on students' 
problems and needs. Based on a domain-referenced approach (Hlvely et 
al, 1973; Baker, 19/4; Popham 1980), the study examines factors which 
may be dlagnostlcal ly useful 1n characterizing or profiling students 
performance across a range of content areas -or domains I.e., factors 
which may be used to structure the test domain, which predict and 
conceptually define Item difficulty? and which likewise may be used to 



structure instructional treatments. The study also explores methods 
for analyzing and structuring diagnostic tests so that the process is 

o 

efficient and conservative of the information load on teachers. 
Specifically the study addresses four inter-rel ated, questions: 

1. What factors ought to be considered in specifying a test 
domain so that the resultant test will provide specific 
Instructional ly relevant diagnostic information? 

2. What analyses procedures can be used to optimally structure a 
diagnostic test to provide valid, reliable, and efficient 
profiles of individual and group performance? 

3. What additional information can be gained from an analysis of 
students 1 wrong answer choices? Can the diagnostic value and 
efficiency of the resultant test be increased? 

4. Are the subject strategies feasible for classroom use, or do 
they require a unrealistic investment in time or an 
impractical level of detail? 

In order to address these questions, we developed an illustrative 

\ " 

test of pronoun use representing factors of interest; administered the 

j 

test to a heterogeneous sample of sixth grade students; used 
general izability theory to analyze results and suggest an optimal test 
structure; examined the consistency and implications of students' 
distractor choices; and finally reflected^ on the entire prbcess to 
assess feasibility and implications for practice. In the sections, 
which follow, we first describe the domain referenced framework which 
guided the test development process and the specific factors which 
were chosen for scrutiny, followed by a description of the test, the 
analytical approach, and the results of our inquiry. 

A Domain Referenced Approach to Test Design 
A domain referenced approach to test design starts with the 
assumption that the major purpose of testing is to assess ah 
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individual's status with respect toya skill or knowledge domain and ^ 

further that valid assessment of that status requires a thorough 

understanding and specification of the domain to be assessed. The 

objective of assessment, in other words, 'heeds to be well defined to 

assure that a test actually measures what it is intended to measure 

and that items reflect test content. The definition is reflected in a 

domain specification which provides a blueprint for developing test: 

items and can serve also to target effective instructional sequences. 

While' a number of approaches to- domain specification have been 

proposed, all seek to define a pool of items that represents an 

important universe of knowledge or skill domain such that student 

performance in one set of items drawn from the domain would generalize 

to a second set of items and. to the entire domain. In its most highly 

< • .• 
prescribed form, domain specifications- provide an exhaustive set of 

rules for generating a set of test items (Hi vely et al , 1973; Osburn, 
1968; Millman, 1980). As more commonly practiced (and as exemplified 
in the present study) domain specifications provide a conceptual map 
of the skill to bej assessed, including relevant parameters for 
defining the range) of eligible content, the response level to be 
represented in the item, item format, directions, and a sample item 

-( Baker; -19747-Pophamr-1980r-Hambl eton^-W80K- ; 

' Regardless of approach,, the Identification of relevant parameters 
becomes a central problem. Establishing content limits Is an initial 
concern, most commonly solved by reference to- extant curricula^ 
material, subject area specialists and/or mutually agreed upon goals 
and boundaries, -- or more preferably research on the structure of the 
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knowledge base and the nature of learning and development. 
Establishing response limits, including criteria for judging 
constructed responses and rules for generating incorrect alternatives 
in selected responses, fixes attention on the quality of expected 
performance, the level of response differentiation desired and. 
systematic 1 error patterns that may be operable. Framed by linguistic 
complexity, form of content, an<j cognitive complexity, test 'content is 
specified to represent, the domain of-interest (see Baker and Herman, 
1983). 

In addition to identifying content parameters which must be 
included -to assure that a test provides a representative picture of a 
skill, diagnostic tests present the additional problem of Isolating 
factors which Influence variations *1n student performance and predict 
varying levels of skill proficiency. In other words, what Important 
factors within a domain 'cause , an item to be more or less difficult or 
a student's performance to vary. Items representing these factors 
then can be appropriately sampled to produce a test with diagnostic 
utility, i.e., one which Identifies the causes or reasons for students 
performance 1 evel . 

What variables might be useful for constructing such diagnostic 
profiles? Research 1n cognitive psychology provides some clues. Chi 
and Glaser (1980) propose a framework for understanding the nature of 
differences between expert and novice performance. Their framework 
characterizes Information processing 1n terms of two components: 
knowledge or content structure, and cognitive processes, components 
which are well supported 1n the research literature. Various authors, 
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for example have pointed to the effect of cognitive processing demands 
inherent in a task. Principal distinctions have been made for tasks 
which require storage, association, and retrieval of information 
contrasted with tasks requiring processing of information, including ^ 
subordination, reconfiguration and other adaptive processes (Spiro., 

1980; Quell malz, 1982). 

Beyond their theoretical justification, content structure and 
cognitive complexity are appealing also in terms of their feasibility 
for practical use. Teachers of course are well used to dealing with 
the structure of content (~ at least as their curriculum or 
instructional materials define it), and their coverage of that 
content, as' research (and intuitive logic) amply demonstrate 1s 
strongly related to student test performance. Cognitive complexity, 
while perhaps not in the common parlance of classroom teachers, can be 
operationally defined to be easily accessible to them. 

The present study is derived from the foregoing framework of 
■ domain referenced testing. A diagnostic test was developed to assess 
one skill within the language arts curriculum. The domain and Item 
pool were ^developed to assess the effects on student performance of 
content structure and cognitive complexity to examine their utility- 
for constructing diagnostic profiles. ThTtesTdeveTopment process 1s 
described 1n the section which follows. 

Development and Administration of the Test 

Design of the Test 

After selecting language arts as a target area for test 
development, local teachers and administrators were asked to Indicate 

15 
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the kinds of grammar problems their students most frequently exhibited 
at the upper elementary and junior high school grade levels. One of 
the most common responses was that students have difficulty with 
pronouns particularly in identifying the correct pronoun referent or 
in using pronouns correctly. They also indicated that a diagnostic 
test of pronoun use would be beneficial for their classroom 

instruction. Pronoun use was therefore selected as an appropriate 

*• 

topic for diagnostic test development. 

I " " 

Following the procedures outlined above, language curricula, 

texts and content experts were consulted to specify the test domain. 

Specifications of linguistic properties, e.g. the recurrence and 

complexity and sequencing of the vocabulary and phrases were also ^~ 

included to assure that the language would be clear and comprehensible 

to the test talcer and that the test could therefore be a measure ' 

pronoun use rather than reading comprehension, (see, for example, 

Doehring and Aulls, 1979). Distractor rules were developed 

systematically to reflect common usage errors. The domain 

specification reflected in particular the two. factors selected for 

inquiry. 

The content structure factor . The curricular review showed that 
nominative, objective, (including direct object, indirect object and 
object of the preposition) and possessive pronouns appear most 
frequently. These five types of pronouns (including the three 
objective forms) correspond to rules of grammar, and are called 
pronoun rules 1n this paper. The review further revealed that the 
pronouns corresponding to each rule can also be classified by form, 
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number and person. There are two types of form: relative form (who 
or whom) and non-relative form. Number pertains to singular (she) and 
plural (they). Person can be of three types: first (I, we), second 
(you), and third (he, they). Since items measuring the second person 
would have sounded, contrived to the reader, the test developed here 
included only the first and third persons. 

The cognitive' complexity factor . The two levels of cognitive 
complexity corresponded to whether students had to use the context of 
a reading passage to determine the correct pronoun. In the first 
level,, the student was presented with a single sentence that included 
an underlined noun(s). The student was to select the pronoun to match 
the underline noun(s). In other words, the pronoun referent was given 
and the student need only associate that referent with the correct 
pronoun. In the second, more complex level, students were presented 
with a short paragraph that included a blank 1n the place of one noun; 
students needed to use the context of the paragraph to identify the 
referent that was appropriate to the blank and then select the correct 
pronoun for that referent. The correct pronoun could be determined 
only from elements of the paragraph 1n which the pronoun was 
embedded. Consequently, the test developed here used two levels of 
embeddedness corresponding to two levels of cognitive complexity; 
non-embedded Items (a single sentence) and embedded Items (a 
paragraph) . 

in summary, the test had five- pronoun factors; Including four 
representing content structure-ind one representing cognitive 
complexity: pronoun rule (nominative, three types of objective, 
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possessive), pronoun form (relative, non-relative), pronoun number 
(singular, plural), pronoun person (first, third), and embeddedness of 
the pronoun (single sentence, paragraph). 

Structure of the test . To investigate the impact of each factor 
on test performance, items were generated for as many combinations of 
the factors as possible. For each combination, two parallel items 
were written. The ideal test would have items for every combination 
of the five factors. Since the form, embeddedness, person, and 
factors each had two levels and the rule factor had five levels, a 
complete test would have 80 (2 X 5) combinations. However, for 
several combinations of factors, sensible items could not be written. 
First, non-embedded items could not be written to elicit singular 
first person pronouns (I, me, or my). Second, items testing the 
relative form of first-pers^ . pronouns would have been contrived. 
Third, there exist no relative form of possessive pronouns. Excluding 
these combinations of factors leaves 46 combinations. Since two 
parallel Items were written for each combination, the total test had 
92 Items. The total design of the test is presented in Table 1. 

The analytic approach used here to analyze the test structure 
requires a fully cros id, balance design. Since the design of the 
total test was unbalanced— 34 cells in Table 1 are empty— 1t was 
necessary to divide the total design into three fully crossed, 
balanced designs- to represent all cells in the design. Deslgn l 
represented the combination of five factors: for (2 levels), 
embeddedness (2 levels), rule (4 levels), number (2 levels), and Items 
(2 levels). This design had 64 items. As Indicated 1n Table 1, the. 
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. Table 1 
Design of the Pronoun Test 



Non Relative Relative Pronoun % 

Non-Errbedded Entedded Non*Embedded Enfcedded' "■ 

1st 3rd 1st 3rd 1st 3rd 1st 3rt , 

Person Person Person Person Person Person Person Person 



Rule: 


• 

Sina. Plur* Sino. Plu'r* Sino. Plur. Sino. Plur. Sine Plur. Sino. Plur. Sina. Plur. Sina. Plnr. 


Nominative 


_ a 2 2 2 _ 2 2 2 _ 2 2 _ 2 '2 ,„ 


Direct Object 


2 2 2 _ 2 2 2 2 2 ! 2 2 


Indirect Object 


2 2 2 ^ 2 2 X 2 2 2 2 2 


Object of Preposition 


2 2 2 _ 2 2 2 2 2 _ _ 2 2 


Possessive 


2 2 2 2 2 2 



a No items in this cell. 
\ 2 items per cell, 



» 
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inclusion of the form factor made it impossible to include items 
measuring first person pronouns and items measuring possessive 

I 

pronouns. 

The two remaining designs were formed to include the possessive 
rule. Since the possessive rule applies only to non-relative 
pronouns, these two designs consisted only of non-relative items. One 
design (Design II) incorporated the contrast between singular and 
plural pronouns (number). The other design (Design III) incorporated 
the constrast between first person and third person pronouns 
(person). Design II, then, included four factors: embeddedness (2 
levels), rule (5 levels), number (2 levels), and Item. (2 -levels) , . 
resulting in 40 items. Design III also included four factors: 
embeddedness (2 levels), rule (5 levels), person (2 levels), and Item 
(2 levels), resulting in 40 items. Many items in the test were 
included in more than one of the three designs. All of the analyses 
presented in this paper focus on these three designs. 

Structure of the item . The test used a multiple choice format 
with five responses per Hern. Three dlstractors were correct 1n all 
ways but one. The fourth dlstractor was correct 1n only one way or 
not at all. An example 1s the following Item, "Mom praised Mary and 
~SteWe" 7~wTtn"~the ~f ol 1 owl ng responses : them, they, us, him and she. 
The correct response (them) 1s an objective, plural third-person* 
pronoun. The next three responses (they, us, and him) were correct on 
two of the three factors (rule, number, person). The final response 
(she) was correct only in the person. The last response was 
considered a "wild card" dlstractor (a highly unlikely selection). 
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Such dis tractors were included to detect guessing or carelessness. 
Test Administration 

Through pilot administrations and feedback from teachers and 
students, the test was modified three times. The final diagnostic 
test was administered to 128 sixth-grade students from three 
elementary schools within a local inner-city district. These schools 
are located in a low to middle SES area with a high rate of transition 
and a mixed population. Approximately 90 percent of the students were 
of Hispanic background, 6% were Black, 2%. were Asian, and 2% were 
non-minority Whites. There were 79 students classified as FEP (Flut-nt 
English Proficient) and 49 classified as LEP (Limited English 
Proficient). Language classification was indicated by the district, 
based on district reclassification criteria of language proficiency 
tests, achievement tests and teacher judgment. 

Two forms of the diagnostic test were prepared. Both contained 
the same items but the order of the items was inverted: Items that 
appeared on the first half of Form A were placed on the second half of 

Form B and vice versa. 

Staff researchers were trained to administer the test. The test 
Instruct ions allowed the administrators to clarify the meaning 1n 
vocabulary Item stems but not in Item dlstractors. The tests were 
-administered at the schools. Students were allowed up to 90 minutes ^ 
to complete the test although most students finished t&e-test 1n about 
45-60 minutes. Classroom teachers were, presented during testing. 
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Overview of the Analytic Approach 
The test score 'that a teacher uses to evaluate students' grasp of 
a curricular unit 1s typically the total score. If the whole class 
does poorly on a test of fractions, the teacher may decide to spend 
more time on the unit. If some students in the class do poorly on the 
test, the teacher may provide them with remedial instruction. 
Traditional approaches to reliability in educational and psychological 
measurement concern the dependability of that total score. The 
approaches focus on the consistency of students' scores over time 
(test-retest reliability) or from one test form to another (parallel 
forms reliability), or focus on the consistency of students' 
performance across items or sections of a test (Internal consistency ( 
reliability). 

Traditional Approach to Internal Consistency 

Of the traditional approaches to reliability, only Internal 
consistency reliability addresses the variability of performance 
across Items within a test, internal consistency alpha, for example, 
Indicates how consistent student performance is across all Items 1n a 
test. The magnitude of the coefficient shows whether the 
rank-ordering of student performance 1s stable across all items. A 

hTgtrvalw - ot^Tptra-fatnornwaT^lT^ 

perform. better than other students on one Item also do so on the other 
items. A low value of alpha (at or near zero) Indicates that the 
students who perform best on some Items are not the same students who 
perfor best on other Items. The latter result suggests that all Items 
on the test are not measuring the same construct, and that student 
performance 1s different across different parts of the test. In this 
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situation, the total test score is probably a poor indicator of 
students' mastery of the material. 

While traditional approaches to internal consistency reliability 
provide some information about the consistency of performance across 
items in a test, they have limited usefulness for diagnosing specific 
areas of difficulty. For diagnostic purposes, it is important to have 
information^about student performance on different parts of the test, 
i.e., a profile of scores. In the test of pronouns developed in the 
current study, it would be possible tv. obtain separate scores for each 
each rule of speecn (nominative, obj^-t e, etc.) , for singular and 
plural items, for first and third person items, and for each form of 
item (embedded in multiple sentences or non-embedded). While it would 
be possible to obtain such a detailed profile of scores for each 
student, this level of detail may not be necessary and might not be 
worth the cost of obtaining it. The central question is what level of 
detail in a profile is necessary to inform a teacher about 
difficulties that individual students or groups of students are having 
with the material. 
The Analytic Question s 

The analytic approach u sed in the present study focuses on the 
consistency of students' performance across multiple dimensions of a 
test, each dimension designed to measure a different aspect of the 
curricular unit. The aim of the analysis 1s to determine the minimum 
amount of Information about student performance on. the test that needs 
to be presented to guide teachers' future instructional decisions for 
Individual students or for groups of students. The analysis addresses 
three issues: (1) the necessity of comgl£l ng° prof 11 es of scores for 
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individual students rather than only one for the class (or one for 
each subgroup of students in the class), (2) the level of detail that 
is necessary in the group or individual profiles, and (3) the number 
of items that are needed to obtain reliable scores in a profile. 

Regarding the first issue, if all students have difficulty with 
the same material (for example,. all students misunderstand how to use 
possessive pronouns), then a single profile for the whole class may be 
sufficient for diagnosing areas of difficulty. If some material is 
particularly troublesome to some students but is not troublesome to 
other students, then profiles for individual students may be 
necessary. Regarding the second isssue, if students perform equally 
well on all rules (nominative, objective, possessive), then it would 
not be necessary to provide separate scores for each rule. If, on the 
other hand, mastery of nominative' pronouns is much greater than that 
of possessive pronouns, then it would be necessary to include in the 
profile separate, scores for each rule. Regarding the third issue, 
once it is determined what scores shoultl compose a profile, the 
question- remains about the number of items that are needed to reliably 
measure each skill represented in the profile. 

Multidimensional Approach to Test Structure: General i2ability Theory 
- \ Sketch of general izability theory . To address the above issues, 
performance on the pronoun test was analyzed using general inability 
thfc General izability (6) theory is a measurement theory designed 
to assess multiple sources of variation in% measurement (see 
Cronbach, Gleser, Nanda, 4 Rajarantnam, 1972; Shavelson 4 Webb, 1981; 
Webb 4 Shavelson, 1981). In a nutshell, G theory uses analysis of 
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variance to partition sources of variation in measures of performance 
of behavior. The results of a general izabil ity study show the 
relative magnitudes of the sources of variation in a test and can be . 
used to improve its design. 

A measurement is a sample from a universe of admissible 
observations, characterized by one or more sources of error variation 
or facets (e.g., items, rules of grammar). This universe is typically 
defined as all combinations of the levels (called conditions, in G 
theory) of the 'facets. Since different measurements may represent 
different universes; G theory speaksNpf universe scores rather than 
true scores, acknowledging that thereAre different universes to which 
decision makers, may generalize. Likewise,' the theory speaks of 
general izabil ity coefficients rather than the reliability coefficient, 
realizing that the value of the coefficient may change as definitions 

of the universe change. 

In G theory, a measurement is decomposed into a component for the 
•universe score and one or more error components. As an illustration, 
consider a 10-1 tern test of pronoun knowledge in which 5 items measure 
singular pronouns and 5 items measure plural pronouns. This test has 
two facets: pronoun number (singular vs. plural) and item. If 20 
students take this test, then the design underlying this study is a 
two-facet partially nested design with items (i) nested within pronoun 
number (n) and crossed with student <$). The object of ^measurement, 
here students,' is not a source of error and, therefore, is not a 



facet. 
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The variance of the observed scores on this test (over all 
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students and al litems for each pronoun number) can be decompsed into 
independent sources of variation due to differenced between students, 
items, and pronoun number and the interactions among .them using 
analysis of variance. , From the analysis of variance, an estimate of 
each component „of variation in the scores is obtained: 

V V ; lni • °s> and °si ,sni .V f 1tems are nested 

within pronoun number in this design, trie main effect for item (1) is 

confounded with the interaction between item and pronoun number (nD.) 
G theory focuses on these variance components. The relative- 
magnitudes of the components provide information about particular 
sources ofjtariation influencing performance on the test. The 
estimated variance component for- students, a s ■■ ', 1s the universe 
score variance and is analogous to the true score variance 1n 
classical theory. The remaining variance components are considered 

error components. . . 

& theory recognizes that decision makers (teachers, for example) 
nfay use the same score in different ways." Some interpretations focus 
on individual differences (relative decisions). For example, the 
teacher may be concerned mainly with the general tzabll 1ty of the rank 
ordering o¥ students, 1n order to give remedial Instruction to the ten 
lowest-scoring students. Other Interpretation's may focus . on the level 
of student performance Uself, without reference to other students' 
peformance (absolute decisions) . For example, the' teacher may be . fj 
concerned* about a student's absolute level of pronoun knowledge, not 
how well he or she .does relative to other students 1n the class. 
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Measurement error 1s defined differently for each of these 
proposed Interpretations. For relative decisions , the error variance 
consjsts of all variance components representing Interactions with the 
object of measurement (here, students): 



"2 "2 

"2 _ ^sn + g si ,sni ,e 

• ff Rel " n„ n.n„ 

n ..in 



In the above equation, n n 1s the number of levels of the pronoun- 
number facet and m 1s the number of Items per pronoun number. The 
error variance for relative decisions reflects differences 1n rank 
orderlng'of students across Items and pronoun number. Ifan 
Interaction effect 1s large, then students' scores are not rank 
ordered the same across levels of the fact. For example, 1f the 
component representing the Interaction between students and number 1s 
large relative to the other components, then students who perform the 
best on singular Items are not the same students who perform the best 
on plural Items. 

For absolute decisions , the error variance consists of all 
variance components except that for universe scores: 

•*2 A 2 "2 "2 

*2 _ ^n_ , q 1,n1 , q sn. , g s1,sn1.e 



°Abs " n_ T n,n n n n T n^n 
n in n in 
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The error variance for absolute decisions reflects differences in mean 

performance of students across items and pronoun number as well as 

differences in rankings. of students. When the decision maker is 

concerned with the absolute level of student performance, the variance 

components associated with effect of pronoun number and items ( <? n 

and o? ■) are included in error variance. The difficulty of one 
i f n i 

item as compared with another will influence a person's score. A test 
composed of easy items will suggest a higher level of proficiency than 
a test composed with difficult items. A large component for pronoun 
number, as another example, indicates that students find items one of 
number (say, plural) more difficult than items of the other number 
(singular^. 

Generali2ability theory and score profiles . The relative 
magnitudes of the variance components contributing to relative error 
variance and absolute error variance can be used to determine what 
kinds of score profiles are necessary for diagnostic purposes. 
Wherever variance components contributing to relative error variance 
(interaction with students) are large, separate profiles are necessary 
for diagnosing learning difficulties. If the interaction between 
students and pronoun number is large, separate profiles would show 
which students were having more difficulty with plural items than 
singular items and which students were having more difficulty with 
singular items than with plural Items., If the variance components 
contributing to relative error (interactions with students) are small, 
but the remaining components that contribute to absolute error 
(components that do not involve interations with students) are large, 
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then one profile for the class would be sufficient. For example, if 
all students find plural Items more difficult then singular Items (a 
TaTgVvarTance component for pronoun number, ), than a 

profile for the class (the means for singular items and plural Items) 
would show the average difference between plural and singular Items. 
Finally, 1f the variance components that contribute to relative error 
variance and absolute error variance are both small, then student 
performance does not, vary across the dimensions of the test. In this 
case, the total score on the test would be sufficient to guide 
decisions about instruction. 

The above description concerns the relative magnitudes of the 
variance components, that 1s, the proportion of total variance 
accounted for by each variance component. A difficult decision 1s 
what proportion to be considered large. There is no rule of thumb 
about what proportion should.be considered large. In the present 
study, all variance components that account for at least 3.5 % of the 
total variation will be noted and discussed. This level is 

7 

conservative; other researchers might set a level of 5% or even 10% a 
the minimum proportion that should be used. As in all decision 
studies, there is a trade-off between cost and efficiency and 
Information. Using a small proportion as a minimum may produce more 
detailed profiles than are necessary. Using a large proportion as a 
minimum, on the other hand, may cause Important sources of variation 
to be overlooked or disregarded. 

The optimal number of Items 1n a profile . While stressing the 
Importance of variance components and error variances, 6 theory also 
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provides a coefficient analogous to the reliability coefficient in 
classical theory. The general Izability coefficient for relative 

-dec-i si ons-i s-def-1 ned-as^ — ■ - 

-2 

p 2 - °s _ 
Rel -2.-2 
°s °Abs 

An analoguous coefficient can be defined for absolute decisions: 

-2 




P 2 _ r 

°s °Abs 

-2 

The general 1zability coefficient, p , indicates the proportion 

/ A 2 A 2 A 2 "2 \ 

of observed score variance (o s + oR e -| or o s + o^ s ) that 

~2 

is due to universe score variance (o s ) . As the number of 

observations per student Increases (for example, the number of items), 

\ ,*2 "Z \ 

the error variance la or decreases and the 

general Izability coefldent (p ) Increases. 

In the present context, the general IzabHlty coefficient 1s 

useful for determining the number of Items needed to provide a 

general Izable measure of each score 1n a profile. If the relative 

magnitudes of the variance components show that separate scores are 

needed for each student for plural pronouns and for singular pronouns 

(Indicated by a large Interaction between students and pronoun 

number), then one general 1zab1l1ty analysis would be performed for. 

plural Items and another one would be performed for singular Items. 
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The design of each general liability analysis 1s simple: student 

crossed with item. This design has three variance components: one 

? 2 
for students (03) , one for items (c.j) and one for the interaction 

between students and items plus unexplained residual variation 

o 2 . The error variance for relative decisions is: . 
si,e 

a 2 

Rel n. 



and the error variance for absolute decision is: 



Abs n i n. 



If the analysis shows that a suitable level of general 1zab111ty ; (say, 
.70) can be obtained with 10 items, then the test would Include 10 
plural pronoun items and a student's mean on these 10 items would 
constitute his or her score for plural items in the profile. 

If the variance components indicate that a profile of group mean 
scores 1s appropriate, then the object of measurement 1s the group, 
not the student, and the analysis changes accordingly (see Kane A 
Brennan, 1977). In the Illustration used in the present study, there 
are two groups of students defined by their language backgroud: 
fluent English proficient and limited English proficient. In 
determining the mean score for a language group, the object of 
measurement 1s the language group. So the estimated variance 



component for language background (a^) 1s the universe score 
variance* The variation among students Is error variation and so 
student becomes a facet of error* The design of the general izabiHty 
analysis of the number of Items needed to measure a score 1n the group 
mean profile 1s students (s) nested within language group (1) and 
crossed with Item (1). The error variance for relative decisions 1s: 

-2 ~2 
~2 - q s,s1 + q si,sli,e 
a Rel • n g - n $ n i 

and the error variance for absolute decisions 1s: 

"2 "2 A 2 * 2 
-2 . q i v q H :t q s,sl q si,sTi,e 
q Abs n. * n $ n $ n 1 

If the analysis shows that 10 Items are needed to produce a dependable 
measure of the group's knowledge of plural Items, then the test should 
have 10 plural Items* 

Sunknaty . In sunmary, the issues of the appropriate score 
profiles for diagnostic purposes and the number of Items needed to 
produce dependable measures of each score in the profile are addressed 
1n two stages* The first stage 1s a general liability study of- the 
structure of the test* In the Illustration presented 1n this paper, 
the facets Include: rule of grammar, pronoun number, context 
(embedded vs* non-embedded) t person (first person vs. third pet-son, 
form (relative vs. non-relat1ve), and Item. The relative magnitudes 
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of the variance components Inthls design show which score should be 
Included 1n Individual student profiles. The second stage 1s a 
separate general liability analysis for each skill in the individual 
and group profiles to determine the number of Items that should be 
Included in the test to obtain a dependable measure of those skills. 

Results of Illustrative Analyses 
This section Illustrates the analytic approach to diagnostic 
testing described inthe previous section. It summarizes (1) the 
preliminary analyses to determine which population subgroups to - 
Include 1n the general liability analyses; (2) the three designs 
underlying the general liability analyses of test structure; (3) the 
variance components produced by the general 1zab1l 1ty analyses, (4) 
example diagnostic profiles; and (5) the numffer of ' items that would be 
needed to yield dependable measures of each score In the diagnostic 
profiles. . 

Preliminary Analyses 

The first step In the approach to diagnostic testing presented 
"here 1s to determine whether there are distinct population subgroups 
1n the design. In the present illustration, the pronoun test was 
administered to students from multiple classrooms and schools, and 
students differed 1n ethnic background, language background, and age. 
Therefore, preliminary analyses were conducted to determine whether 
these factors influenced performance on the pronoun test.. Analysis o 
variance F tests revealed that the only population characteristic 
Influencing performance on the test was language background (FEP vs. 
LEP; F(l) « 30.09, p < .001). The statistical tests for classroom, 



school, ethnic background, and age were not significant (F statistics 
ranged from .12, p < .73 to 1.06, p < .37. In all further analyses, 

~then~,~ only the" distlnctTon" 

maintained. 

Summary of the Three Designs 

As was described 1n the section sunmarlzing the design of the 
test, the entire test can be described by three.crossed designs. 
Design I 1s a five-facet design yielding 64 Items: embeddedness (2 
levels), pronoun form (2 levels), rules (4 levels), number (2 levels), 
and 2 Items for each combination of the previous four facets. Design 
II 1s a four-facet design yielding 40 Items: embeddedness (2 levels), 
rules (5 levels), number (2 levels), and 2 Items for each combination 
of the previous facets. Design II 1s also a four-facet design 
yielding 40 items: embeddedness (2 levels), rules (5 levels), person 
(2 levels), and 2 Items for each combination. * 

As a result of the complexity of each design, the nuaber of 
variance components 1n each analysis was very large * For example, 1n 
the analysis of Design I, with students nested vithln language 
background and students and language background crossed with 
embeddedness, rules of grawnar, number, form, and Item, there were 51 
variance components. Rather than present the descriptive results 
(means, standard deviations) for a VI variance components 1n each 
design, descriptive results are presented only for components that 
account for at least 3.5% of the total variation 1n the design. Table 
2 presents the variance components that exceed 3.51 of th$ total 
variation 1n each of the three designs. Each number 1n Table 2 
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represents the percentage of total variation accounted for by each 
variance component. Tables 3, 4, and 5 present the means and standard 
deviations corresponding to all variance components listed in Table 
2. The means are the percent correct, so the maximum score possible 
is 1.00. 

Variation due to student and language background. The large 
component for language background in each design indicated that FEP 
and LEP students showed different levels of performance on the test. 
'As the descriptive results in Tables 3 through 5 show, FEP students 
showed higher mastery of pronoun usage than did LEP students. The 
large variance component for students (nested within language 
background) in all three designs shows that there were substantial 
individual differences between students within a language group. Some 
students had mastered pronoun usage while others had not. The 
^component for students, then, reflects the range of mastery of pronoun 

usage in the same. . 

■ ■ ' . \ 

Variation contributing to absolute error . Most of the variance 

components presented 1n Table 2 do not involve interactions with _ 

students or with language background. In Design I, the pronoun form 

facet accounts for the greatest variance (34.0%). Students found 

relative pronoun items to be very difficult. In fact, as can be seen 

1n Table 3, LEP students performed at about chance level on all 

relative pronoun items except those measuring the nominative rule 

(with 5 response choices for each item, chance level 1s 20%). 

Table 2 also shows a substantial effect for the- context of the 

item, whether the sentence was embedded with a paragraph. The 
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Table 2 

Proportion of Total Variation accounted 
For by Each Variance Component 



Variance 

IsUllipUlIC 1 1 I* 


Design I 


Design II 


Design I 


•■ 

Language Background [L] 


12.3 


19.6 


in a 


Student [S(L)3 


20.6 


33.6 


37.3 


Pronoun Form [F] 


34.0 






Embedded [E] 


4.1 


26.2 


17.3 


Rule [R] 


<3.5 


<3.5 


6.0 


F E 


5.1 






F R 


4.6 






F S(L) 


4.0 






E S(L) 


<3.5 ; 


6.6 


<3.5 


Residual 


3.7 - 


5.6 


6.1 


All others 


11.6 


8.4 


13.9 


Total 


100.0 


100.0 


100.0 



a Not applicable. 

Note : Only variance components accounting for more than 3.5% of total 
variance are listed here. 
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variance component for embeddedness is smaller for Design I than for 
Designs II and III because the effect of relative pronouns (who-whom) 
overwhelmed that of embedding 1n Design I. The means 1n Table 3, 4, 
and 5 show that all students found it much more difficult to determine 
correct pronoun usage when the target sentence was embedded within 
other sentences. The difference 1n performance between embedded and 
non-embedded Items was similar for FEP and LEP students. 

Interestingly, the rule of grammar produced substantial variation 
in performance only 1n Design III. As Table 5 shows, students tended 
to perform worse on the Items measuring the possessive rule than on 
Items measuring the other rules. This effect appeared only when Items 
measuring plural pronouns were Included 1n the analysis (Design III), 

« I - 

and. not when singular Items were Included (Design II). As 1s 
Indicated by the small , variance component for rule 1n Design I (where 
the possessive rule was not Included), student performance did not 
vary much across Item measuring knowledge of the nominative and three 
. objective rules; 

Table 2 shows two other effects in Design f that contributed to 
absolute error variance. The pronoun form facet interacted with the 
embeddness facet and with rules. The interaction between pronoun form 
and embeddedness indicates that the difference between performance on 
embedded and non-embedded items was not constant across relative and 
non-relative pronoun Items. This result is clearly seen 1n Table 3. 
Both FEP and LEP students did much better on non-embedded Items than 
on embedded itoms only when the pronouns were not 1n the relative 
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Table 3 

Descriptive Results for Major Sources of Variation 
- - -i n~ Design- 1 - 



FEP a LEP a 
Factor W ' 5D~ W 



Non Relative Pronouns 


.70 


.15 s , 


.54 


.18 


Context 
Non -Embedded 


.87 . 


.17 


- 

.66 


.24 


Fmbedded 


.53 


' .22 


.42 


.20 


RiiIp 

l\Ulc / 

Nominative^ 


.59 


.21 


.49 


.19 


Direct (jbject 


.66 


.23 


.51 


.21 


Indirect Object ' 


.76 

V 


.23 . 


.56 


.28 


Object of Preposition 


' .78 


.18 - 


.60 


.29 


Relative Pronouns 


.34 


.15 


.24 


.13 


Context 
Non-embedded 


.35 


.17 


^.27 


•I 4 


Embedded 


.33 


. -18 


.22 


.15 


Rule 
Nominative 


.57 


,24 


.52 


.24 


Direct Object 


.20 


.18 


.13 


.17 


Indirect Object 


.25 


.21 


.19 


•21 


Object of Preposition 


.33 


-v28 


x V 4 


, .20 



a FEP « Fluent English Speaking. LEP » Limited English Speaking. 
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Table 4 

Descriptive Results for Major Sources of 
Variation in Design II 



FEP LEP 
Factor W HE E \ IE 



Context <-« o/t 

Non Embedded .84 .17 .62 .24 

• 

Embedded .51 .21 .38 .18 



Table 5 

Descriptive Results for Major Sources of 
Variation in Design III 



Factor 


M 


FEP 


M 


A 

l£p 

<* 


Context 






.61 


> 

.23 


Non -Embedded 


.78 


.17 


Embedded 


.56 


.20 

t 


.42 


.19 

3 


Rule 






.49 


.23 


Nominative 


.62 


.20 


Direct Object 


.68 


.22 


.53 


.21 


Indirect Object 


.77 


.23 


.64~ 


.25 


Object of Preposition 


.81 


.20 


.63 


.31 


Possessive 


.46 


.25 


.27 


.21 



< 
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form. When the items called fcr relative pronouns, 1n contrast, 
student performance was very similar. for non-embedded and embedded 
Items. Thus, for relative pronoun items, the presence or absence of 
context did not affect student performance. Students performed poorly 

in both cases. i 
The interaction between pronoun form and rules can also be seen 
clearly 1n Table 3. For non-relative pronouns, students showed' 
similar performance on all rules, with performance on nominative items 
somewhat lower than performance on objective items. For relative 
pronouns, on the other hand, performance on the nominative 1tems"was 
much higher than -performance on the objective items. About half of 
the students knew when to use "who" (the relative form of nominative 
pronouns) but very few FEP students and no LEP students knew when to 
use "whom" (the relative form of objective pronouns). To determine 
whether students perf ormancer di f f ered across the three objective 
rules, Design I was also analyzed without the nominative rules 
(including only the three objective rules). The interaction between 
pronoun form and objective rules nearly disappeared (it accounted for 
only about 1%. of the total variance), showing that student performed 
nearly the same on the three objective rules. Given this finding, 
then, 1t would not be necessary to retain information on the three 
objective rules. The mean for all objective Items as an 
undifferentiated set would be sufficient. 

Variation contributing to relative error . A notable feature of 
Table 2 1s the lack of interactions between any facet and language 
background. This finding shows that the pattern of performance across 
the dimensions of the test among FEP students was the same as that for 
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LEP students. Coupled with the large component for language 
background, this result indicates that the profiles for the two groups 
have the same shape, with the profile for FEP students being higher 



There were surprisingly few interactions between students and 
facets* The component for the interaction between students and 
pronoun form in Table 2 indicates that o the rank order of students on 
relative items was not the same as the rank order of students on the 
non-relative items* There are two possible interpretations of this 
result. The first, wKich is highly unlikely given the huge main 
effect for pronoun form, is that some students fourid the relative 
pronoun item easier than the non-relative pronoun items while the rest 
found the non-relative pronoun items easier than the relative pronoun 
items* A far more likely interpretation is that the difference in 
performance between relative and non-relative pronouns was larger for 
some students than for others*, It is unlikely that any students 
performed better on relative pronouns than on non-relative pronouns. 

similar interpretation can be given for the interaction between 



is that the difference in performance between_embedded and 
non-embedded items was larger for some students than for others. 



Finally,, it should be noted that the residual variance component , 
represents the interaction between all facets the design, including 
students, and language background, plus unsystematic error. A large 



than that for LEP students. 



students and the- embeddedness facet in Design II. Since it is 
unlikely that any student performed better on embedded items than on 
non-embedde'd Items, the most likely Interpretation of the Interaction 
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residual variance component ysually reflects sources of variation that 
have not been taken into account 1n the measurement. The small 
magnitude of the residual component. in all three designs 1n the 
present study suggests that all important test facets have j>een taken 
Into account 1n the design of the test. 

^ The Primary Sources of Variation and Example Diagnostic 
Profiles ' 

The only sources of variation 1n test performance that exceeded 
3.5% of the total variation were the pronoun form, embeddedness, and 
rule facets. The person (first vs. third) and the number (singular 
vs. plural) of the pronoun did not produce variation among students' 
test scores. That is, students "showed equal mastery of first and 
third person pronouns and showed equal mastery of singular and plural 
pronouns. Furthermore, the effect for Items was very small, 
indicating that students performed similarly on both items 1n each 
cell of the test design. . % 

The findings protrayed in Table 2 and described above can be used 
to^make reconmendatlons about the optimal diagnostic profiles for 
pronoun usage for the sample in this Illustrative study^ Only the 
large 'effect contributing to relative error (those involving 
interactions with students) would. need to be ^Incorporated Into the 
score profiles for Individual students. Since only the who-whom and 
embeddedness facets Interacted with students,. the profile for 
Individual students would need only to consist of the mean scores for 
relative pronoun Items, non-relative pronoun Items, embedded Items-, 
and non-embedded Items. Example profiles for three randomly selected 
students appear 1n Figure 1* 



The remaining large 1 variance components— those contributing to 
absolute error but not relative error (components that do not Involve 
Interactions with students)— would guide the formation of class or 
group profiles. Since pronoun form Interacted with embeddedness and 
rules, the group profile should present the means for embeddedness 
(embedded, non-embedded) and rules (nominative, objective, possessive) 
separately for relative Items and non-relative Items, as was done in 
Table 3. Figure 2 presents such profiles for FEP and LEP students. 
Since performance was similar across the three objective rules, only 
the mean socre 1s presented for objective Items. Furthermore, since, 
performance was similar across person' and number, first person plural, 
third person singular, and third person plural Items were combined. 

The mean profiles in Figure 2 show the general patterns of 
performance 1n this sample. Since the rule facet 1n the design did 
not Interact with student, the means for nominative, objective, and 
possessive Items are good representations of the performance of all . 
students. This profile would show that all students need further 
instruction on the possessive rule and the objective rule for relative 
form. Similarly, the general pattern for embeddedness 1n Figure 2 
suggests that students would need further instruction on all embedded 
^pronouns and non-embedded relative pronouns. 

In summary, the variance component analyses 1n the present study 
show that individual profiles for embedded and non-embedded and 
relative and non-relative pronouns, and a group profile for rules of 
grammar would be sufficient for diagnosing Individual and group 
difficulties with pronoun usage. These profiles would be more 



FEP 



LEP 



4- 



4- 



4- 



4- 



4 



Embedded 



Embedded Nominative Objective Possessive Non 



Embedded Nominative 



MM 



Non Relative 



Relative 



Figure 2, Mean Profiles for FEP and LEP Students 
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Informative than the total score of the test, and suggest that 
diagnostic decisions based only on the total score might lead to 
erroneous consequences for the student and for the class. Not only do 
the variance component analyses shovv ^rch aspects of test performance 
should be tabulated for individual and group diagnosis, they are also 
valuable for showing which aspects of pronoun usage do not need to be 
tabulated* Since student performance did not vary across number and 
person, these facets could be omitted from the diagnostic profiles. 
The Optimal Number of Items 

The previous section demonstrated how to use the relative 
magnitudes of the variance components to guide slectlon of scores for 
student and group profiles. This section reports the results of 
general liability analyses to show how many items would have to be 
included 1n the test for dependable profile scores. The design of the 
general liability analysis for each of the four scores In the - 
individual profile (non relative, relative, non-embedded, embedded, 
see Figure 2) was students crossed with items. The items used in each 
genera! liability analysis were all Items in the original test that 
pertained to that score. The general liability analysis of 
non-embedded Items, for example, Included all first-person, third- 
person, singular, ^plural, relative, non-relative, nominative, 
objective » and possessive items that were non-embedded. 

The results of the general liability analyses are presented 1n 
Table 6. Table 6 shows the number of Items corresponding to different 
levels of general lability. For example, it would take at least 10 

50 ' / ' .■ 
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TABLE 6 



Number of Items Corresponding to Different 
General izability Coefficients 



Score 



Relative Decisions 
.50 .60 .70 .80 



A^cnlutP Decisions 
.50 .60 .70 .80 



INDIVIDUAL PROFILE 

Non- Embedded 
Embedded 
Non Relative 
Relative 

GROUP PROFILE 

Non Relative 
Non- Embedded 
Embedded 
Nominative 
Objective 
Possessive 



Relative 
Non-Embedded 
Embedded 
Nominative 
Objective 



8 


10+ 


10+ 


10+ 


9 


10+ 


10+ 


10+ 


5 


9 


10+ 


10+ 


10 


10+ 


10+ 


10+ 



1 


1 


1 


1 


1 


2 


3 


7 


2 


3 


4 


9 


1 


1 


1 


2 


1 


_ 1 


1 


1 


2 


3 


4 


9 


1 


2 


2 


4 


1 


1 


1 


1 



.10+ 10+ 1-0+ 
10+ 10+ /10+ 
10^10+ /10+ 

fo+- 




10+ 10+ 

4 7 

2 2 

2 2 



10+ 
10 
3 
4 



10+ 
10+ 
5 

3 



More than 10 items would be needed 

to obtain this level of general Izability. 
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items to measure individual proficiency on non-embedded pronouns with 
a .70 level r.f ger-rali? - !V for rolatr ind absolute decisions. 
On und . nly 3 i „em woul 2ded to measure group mean 

proficiency on non-embedded pronouns at the same level of 
general izability for relative decisions {two items would be needed for 
absolute decisions). 

Since the same items can be used to measure different aspects of 
pronoun usage, the total number of items in the test needed to obtain 
general izable measures of each score in the profile is smaller than 
the sum of the number of items in Table 6. For 0 example, an embedded, 
relative nominative item can be used to measure embedded pronoun 
usage, relative pronoun (who, whom) usage, and nominative pronoun 
usage. If the scores in the individual and group profiles were to be 
used for relative decisions (for example, selecting the bottom 20% of 
students for remedial instruction), a pronoun test of 20 items could 
be constructed so that each score in the profile had at least .70 
general inability. A pronoun test with the following configuration 
would satisfy the requirement listed in Table 6: 4 non who-whom 
objective (embedded) item, 1 non who-whom possessive (embedded) item, 
1 who-whom nominative (non-embedded) Item, 2 who-whom objective 
(embedded) items, 3 other who-whom non-embedded Items, 3 other 
embedded items (all who-whom) and 5 other non-embedded Items (1 
whom-whom and 4 non-who-whom) • For absolute decisions, a pronoun test 
with 40 Hems could be constructed so that each score 1n the 
individual and group profiles had a level of general izability of .70. 
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Dlstractor Analysis 
Test subscores based on simple right-wrong scoring, such as those 
reported above, utilize only a portion of the data that may be 
available from student responses. The nature of their Incorrect 
response provides potentially useful additional Information for. 
diagnosing student needs. In the case of multiple choice Items where 
dlstractors are constructed to represent specific errors or mis- 
conceptions, analysis of dlstractor choices might supply several 

it 

pieces of Information: 

First, such an analyses can point to potential reasons for 
students 1 difficulties with certain subscales by Identifying the kinds 



of errors they made. For example, the dlstractor analysis of the 
relative pronoun Items can show: 

Whether student responses are systematic or random: e.g., did 
they always choose between who, and which, op was there another 
popular response? Here they consistently choosing among these 
dis tractors? 

Whether the kinds>of errors students make are due to a 
-misconception of a rule or due to some other feature. For 
example, if students consistently choose who when whom 1s the 
v correct response,, they do not know the, rule which regulates the 
use of these two pronouns". If there is another frequently 
selected dlstractor such as "he", they may be exhibiting an 
additional problem 1n the construction of independent clauses. 

' Secondly, dlstractor analysis can Indicate error patterns across 
subscales. These patterns can point to erroneous rules, misconcep- 
tions or misinformation which are applicable to several or all of the 
subscales. For example, one of the error patterns that occurred for 
the relative pronoun Items might also appear In Items testing other 
types of pronouns and suggest that students are having difficulty with 
a particular aspect of syntax. 
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Method 

Recall that Item distractors are organized such that three 

. b . 

distractors represent a specified error and one dlstractor reflects 

guessing. Matched pair of item contain the same distractors. For 
example: 

Cornelia is a Very pretty woman. David is a handsome man. 

- him (Incorrect gender & pronoun) her * 

they (Incorrect case) they 

she (Incorrect response) he 
< he (Incorrect gender) she 

her (Incorrect pronoun type) him 

Each 1 tem . was___an 

takers selected each dlstractor; and contingency analyses were 

conducted to examine consistency of student responses. Responses were 

to be analyzed to address a number of enter-related questions: 

1. Were the response patterns within matched pairs of Items 
consistent? That is, did students select the same 
distractors for both items? 

Zl Did students make the same error across several types of 
items or were some errors characteristic of one type of 
item? 

3. He know that the pattern of responses differed for 
non-embedded and embedded Items; can dlstractor analysis 
point to potential reasons for this pattern? 

4. Did Limited and Fluent English Proficient students {LEP & 
FEP) differ in their response patterns? Did patterns 
Indicate that tftey were making different or similar types of 

. errors. 

Results 

the validity of an analysis of dlstractor patterns rest on some 
cpnslstency 1n student wrong answer choices. When dlstractor patterns 



are consistent within a matched pair it seems reasonable to assume 
that the error patterns reflect the misconceptions or misinformation 
specified by the distractors. In the absence of such consistency, 
then it is unclear whether students are selecting at random, are 
reacting to some pecularity in item content or to linguistic or other 
properties. 

Contrary to expectation, contingency analyses found little 

consistency in student wrong answer choices. Distractor selection was 

consistent, on the average, in only 21% of the cases, ranging from -0% 

to 48% over the 42 parallel pairs of items included on the test; 

two-thirds ,of s the items fell in the 11-30% range (See. Table 7) . There 

was greater consistency among more difficult items^, but no apparent 

differences between fully and limited English proficient Students. No 

patterns were^ discernible by pronoun rule. 

Table 7 ,J ' 
Agreement on distractor choices 
. for parallel pairs of items 

Wrong Answers Only" , 

Limited English 
Proficient 



Fluent English 
proficient 



Mean % of agreement 


frequency 


% 


frequency 


« % 


0-10% 


7 


(15) 


"?•* 7 


(15) 


11-20% 


■ 13 


(28)' 


16 


(35) 


21-30% 


19 


(41) 


15 


(33) 


31-40% 


5 


(12) 


6 


(13) 


40-50% 


2 , 


( 4) 


2 


( 4) 




46 pairs 


(100) 


46 pairs 


(100) 



h*49 
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n=79 



Future analyses will examine alternative models for describing 
student error patterns, looking for Instance at whether students are 
consistently eliminating certain dlstractors and guessing at random 
from those remaining. The results of these analyses will be Included 
1n ^ subsequent report. 



***** 
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Conclusions 

This paper described a four-step approach to constructing a - - 
diagnostic test that provides precise but practical information on 
students' problems and needs for additional instruction or 
remediation^ The approach is based on analyzing the structure of the 
domain to determine which skills within the domain need to be assessed 
to diagnose students' problems. 

The first seep in the diagnostic process described .here was to 
identify the factors that described the curricular domain (here, 
pronoun usage). Four content factors were identified: the rule of 
grammar (nominative, objective, possessive), the pronoun form 
(relative— who or whom, non-relative), the number (singular, plural), 
and the person (first, third).- In addition, a factor corresponding to 
cognitive complexity was Identifed: whether the context of the 
reading passage had to be taken Into account to determine the correct 
pronoun.' This factor was operatlonallzed in two levels of embedding: 
a single sentence or a pragraph. 

The second step was to- construct a test with Items representing 
all possible combinations of factors (content and cognitive 
complexity). Sensible Items could be written for 46 combinations of 
factors. Two Items were written for each combination, resulting 1n a 
92. item test. 

The third step 1n the diagnostic testing process used 
general izablHty theory to determine which factors and Interactions 
among them produced variation in students' scores. Specifically, the 
relative magnitudes of the variance components corresponding to all 
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factors and Interaction among them 1n the test revealed which factors 
were Important. This Information was used to Identify the Information 
needed 1n diagnostic profiles. Only two content factors,; rule and 
pronoun form, produced variation 1n student performance* The other 
two content factors, number and person, did not. Furthermore, 
cognitive complexity also had a large ef f ect* on student performance. 

Some difficulties were common to alt students' (e.g., all students 
had more difficulty with possessive pronouns than with objective 
pronouns). This Information could be entered 1n a single profile for 
the group or class. Other difficulties applied to some students but 
not others (e.g., some students did much* worse on embedded Items than 
on non-embedded Items while other students performed similarlly on 
both types of items). This Information would be part of profiles for 
.Individual students. Since the number and person factors had no 
effect on student performance—all students performed about the same 
on singular and plural items on first-person and third-person 
items— there was no need to distinguish between these skills in the 
test or in the profiles. 

Based on the information about the necfessary ingredients of 
diagnostic profiles, the final step in the analytic process was to 
determine the minimum number of Items needed to obtain general izable 
measure of each skill in the diagnostic profile. The results of the 
general izability analyses showed that a 20-item test would be 
sufficient to measure mastery of pronoun usage if the teachers 
interest was in identifying the students with greatest need for 
additional instruction in each skill. A 40-1 tern test would be 
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sufficient ifKhe teacher's Interest was in identifying each student's 
absolute level of mastery of each skill. 

In short, the structure of the domain consisted of 46 skills 1n 
pronoun usage (all sensible combinations of the five factors). The 
1n1tal test consisted of 92 items, 2 per skill. To adequately measure 
student performance on each of these 46 skills would probably take 
between 2 and 10 items per skill, resulting 1n an extremely long 
test. The analyses performed here showed that only 9 of the 46 skills 
need be assessed resulting 1n a vastly simplified and shorter 
diagnostic test. 

Although the .entire process of (1) Identifying a domain, (2) 
constructing an 1n1ta1 test to fully represent the domain, (3) 
analyzing the performance on the 1n1ta1 test to determine the factors ' 
that. Influence student performance, and (4) constructing the final 
optimal test would be too time-consuming for a classroom teacher, the 
use of the final diagnostic test and score profiles would certainly be 
feasible for classroom practice. With a relatively short test 
(maximum of 20 minute test to administer, In this case), the teacher 
could identify students 1 strengths and weaknesses on all Important 
aspects of the curriculum domain and make Instructional decisions 
accordingly. 

Specification of the domain structure underlying the, test 1n an 
Important Issue 1n this diagnostic approach. It 1s Important to 
specify the test as completely as possible. If factors 1n the test 
are left out, difficulties that students have on the test may be 
attributed to the wrong skills or may not be able to be Identified at 
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all. Although complete specification 1s Important, 1t 1s not 
necessarily difficult. In ,the present study, the general 1zab1 11 ty ■ 
analyses showed that only a small amount of variation 1n test 
performance was attributed to unexplained factors. Consequently, 1t 
1s reasonable to conclude that all Important factors 1n the domain 
were Included. 

Also Important 1n domain specification 1s not to restruct the 
domain only to aspects of content. Although several content factors 
did affect student performance, the cognltve cot^;lex1ty of the Item 
had a major Impact on performance. For example, even though many 
students could correctly Identify when nominative pronouns should be 
used 1n a single sentence (a low level of cognitive complexity), many 
of them could no do so when the sentence was embedded 1n a paragraph 
requiring them to use the context of the paragraph (a high level of 
cognitive complexity). A teacher would come to different conclusions 
about mastery of pronoun, usage from a test with items of low cognitive 
complexity and from a test with items of high cognitive complexity. 
Without taking Into account the Influence of cognitive complexity on 
performance, the teacher may well make erroneous decisions about the 
need for additional Instruction. 

Finally, the results of the Illustrative analyses presented here 
also have Implications for taking Into account multiple studei 
populations* Tc^ht-s often give different tests to students from 
different population subgroups (for example, different language 
backgrounds), assuming that the performance of the groups 1s 
different. An Implicit 'assumption, therefore, may be that some 



groups excel on some material while other groups excel on other 
material; that 1s, tha\ profiles of different groups may have 
different shapes. The strikingly .parallel profiles of fluent English 
proficient students and limited English proficient students in the 
present Illustrative study, however, raises a question about whether 
different tests are necessary. In this case, separate tests for each 
group would be unnecessary. To take Into account the mean differences 
1n performance between groups (fluent and limited English proficient 
students), the "items measuring a particular skill on the diagnostic 
test could cover a range .of difficulty (for example, varying the 
vocabulary level, or Tength of the sentences in a item). 
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Diagnosing Student Errors: 
An Example From Science 

Steve Shaha 
Center for the Study of Evaluation 

Most tests of science are designed to assess knowledge of scientific 
facts. Information gained from such tests generally 1s summarized 1n a 
score which reflects the number of correct responses made. Hence, any 
diagnostic Information 1s relatively limited; the number of correct 
responses relates to knowledge of scientific facts alone and not to ah 
Interpretable summary of the nature of a student's misunderstanding. More 
often than not, total scores which are useful for assessing class standing 
are not very useful for helping a teacher to aid Individual students 1n 
overcoming specific problems. Perhaps the most promising feature of 
r diagnostic tests 1s that they will yield Information above the level of 
mere number of correct versus Incorrect responses. Well designed 
diagnostic tests would give specific Information to Instructors concerning 
the Individual student's problems. 

As part of a larger project 1n diagnostic assessment, the Center for 
the Study of Evaluation undertook the development of a prototype diagnostic 
test of understanding of science at the high school level. The test's 
primary purpose was to provide detailed information concerning «itrengths 
and weaknesses 1n student's skills in scientific reasoning. Our interest 
was a) to isolate specific^ errors of scientific reasoning which byrthetr— : 
consistency could be systematically diagnosed, and b) to examine 
differential patterns of error at different levels of comprehension. The 
first factor necessitates a rational choice of distractors; the second 



necessitates an orderly structuring of items by complexity of Item content. 

Previous work in diagnostic testing in science is notable for its 
variety of almost completely independent derivations, in a number of 
countries around the world, of both theories of science understanding and 
tests which measure it. After the well-known but two-decade-old "Test on ^ 
Understanding Science" (Cooley 4 Klopfer, 1961) and the massive "Test of 
Understanding the Mature of Science" (IEA, 1969), there appears to have 
been no new well -documented instrument in this area. However, Independent 
workers have succeeded in certain areas with demonstration projects. 
Dillashaw and Okey (1980), from the^JS, developed a test fcr diagnosing 
five process skills in science. Their findings justified the use of 
diagnostic tests, showing that both achievement scores and attitudes toward 
science rose for high school students when information from diagnostic 
tests was used for remedial purposes. Gorodetsky and Hoz (1980), from 
Israel, explored the use of profile analysis techniques for diagnosing 
conceptual misjudgments in scientific logic. Information from this type of 
diagnostic test showed that certain conceptual problems associated with, 
lags in science learning can be identified, and that remediation of these 
problems can lead to resolution of learning lags.. Dreyfus'and Jungwirth / 
(1980a; 1980b), also from Israel, succeeded in isolating and classifying 
the actual types of errors most often committed in responding to questions 
concerning science. Their efforts showed that is was indeed possible to 
develop diagnostic tests of scientific reasoning based upon the assessment 
of patterns in errors committed. ^ 

Bill eh and Malik (1977), from Jordan and Pakistan respectively, report 
success at the college level with an 85 Trem^instrument-which taps 

\ 
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assumptions of- science, scientific ethics, and -other areas; the results 
might be applicable^ a. diagnostic interpretation but only summary scores 
are presented. '.Rubba & Anderson (1978), from the US, developed an 

o 

instrument based on a nine-factor conceptualization of science undertandlng 
•by Showalter (1974); again diagnostic Interpretations may be feasible but 
summary scores alone are discussed.^ Cantu & Herron (1978), from Mexico and 
the US respectively,, present a science concept attainment test which allows 
an interpretation following a plagetl an understanding of how students learn 
1n that field. Osborne (1978), from New Zealand, explores subscales of a 
test of college-level physics which allow a diagnostic Interpretation 1n 
each of seven currlcular components. Rodrlgues (1980), from Brazil, 
admlniTters^a^set.of cartoons of certain physical situations which reflect 
laws at work, and forms a diagnosis of sorts based on characterizing the • 
narratives children give 1n, response. ° ; 

The best single theoretical contribution to diagnostic testing is by 
Johnstone (1981), from Scotland. He presents some selected examples of 
college level science tests which allow an Interpretation of specific types 
of misunderstanding. He also discusses a variety of distorting factors 
which are germalne not only to science tests but to all educational tests 
from which a diagnosis might be derived. These distortions arise 1n the 
context of superficial testing (and so a test "must probe Into each linkage 
and subconcept as well as take a global picture" p. 39); too much guessing 
(and so a test should allow confidence marking) and issues of language 
clarity. He takes the view that diagnostic tests in science 
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are relatively easy to construct because course objectives can be 
specified for* a major portion of the study* Items can be written to 
test each objective from several angles and the items can often be in 
.an objective format which lends itself to. computer marking, (p. 34). 

The reported ease ? of construction would be a significant advantage if true 
^but it remains to be studied in more detail. 
Test Development 

Because none of the extant tests of science understanding In- 
corporate ;both a rational basis for distractor choice and a uniform 
layering of items by complexity, four topics areas within science were 
selected for a prototype diagnostic test of science: phothosynthesis, 
magnetism, energy in a closed system, "and gravity. The target audience was 
the high school level. It is important to note that since we could not 
control- for instruction received by students tested, we selected areas of - 
science which we considered to be generally covered from seventh grade 
upward, based on a search of commonly used textbooks. 

Item generation for the present study was based on two parallel 
considerations, the definition of content and principles to be tested and 
the determination of the specific types of errors which the test would be 
designed to ..diagnose. On the matter of content, we decided to develop 
items which mirrored the level at which a given content area was understood 
— that is, items which would provide a test of a student's depth of 
comprehension. To accomplish this we chose a three level comprehension ^ 
strategy: , 
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* Low level -- Definition. Items designed to 

test for knowledge of a principle at the * 
definition or factual level. 

* Middle level ~ Principle Application. Items which 
\ required that subjects recognize the use of a 

principle in a given situation or context. 

* High level — Problem Solving, items which required 

" subjects to analyse a problem or probably 
7^ unfamiliar situation and arrive^at. the 

Correct solution by applying the principle in 
question. 

The next ste^was^to define- the errors to be diagnosed. Diagnostic 
tests do. not merely measure correct versus incorrect responding; but are 
specifically designed to assess well-defined errors in a systematic 
manner. The error types we selected for diagnosfflg in the four areas 
chosen were based on the research of Dreyfus and Jungwirth U980). In 
their study, they were successful in isolating five classifications of 
error types which are common in scientific reasoning, of which the 
following errors were of interest in the present study:. 

Logical fallacy — Student's response based upon faulty logic, 
p such as circular reasoning, general izind- beyond the specific 
"(P situation, or imputing cause from correlation. 

Intuition Students response based upon intuitive logic or 
prior experience rather than a understanding of the 
problem. 

Content — Student's response selected due to similarity with the 
language in the stem. Also test wise errors fit in this 
category j sjuch as selecting the most scientific-sounding 
alternative or the longest choice 'available. 

The items generated' for each topic area selected were, based on 

crossing the two dimensions: each level of comprehension to.be tested, 

with each error type to be measured. There were three items generated at 

each level of comprehension, and every error type was assessed three times 

at eacn given comprehension level. Strict procedures were followed in the 
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establishment of domain specifications, rules for generating items, and the 
outlining of objectives to be tested (which are elaborated in the 
accompanying manual by McArthur, Shaha, Choppin, & Hafner [1983]). 
Item Design 

The item design process is summarized below. The most effective *way 
to envision the processMs to consider actual test items. For this 

purpose, three items from the area of Magnetism will be explored, beginning 

I 

with the lowest level of comprehension. 

The first item of interest is the item at the low level of 
comprehension, the item which should be easiest for the most subjects. 
This item was constructed to test the most. basic principles of the concept, 
even at the level -of rote learning. In magnetism, the following item stem 
was developed: 

Nails made of metal alloys (mixtures of metals) are 
attracted by magnets. 

The stem requires only that a student verify the truthfulness or falsehood 
of the assertion presented. In this instance the distractors are a tool 
for sorting out the precise reasons for both correct or erroneous 
responses. Each dlstractor contained a possible rationale for deciding 
whether the statement vas true or false, and these rational^sjrfere designed 
. to provide information concerning the precise type of errors being 
committed. Of the four distractors per item, two allowed, selecting True as 
a response, and two allowed False.. The following were the four 
alternatives for the. item above: t 

71 



- 7 - 



a. True -- because all metals are 

magnetic, [Logical fallacy] 

b. True — only if the alloy contains 

iron. [Correct response] 

c. False --because only iron can be 

magnetized. . [Content distraction] 

• V 

d. False— because nails are not the 

same shape as magnets. [Intuition distraction] 

The first alternative represents a logical fallacy, since selecting this 
alternative indicates that a generalizing from a single example (iron) to 
an entire universe of related examples (metals). The second alternative is 

correct. The third alternative is indicative of an error based on content 

/ ■ 

. / 

because its wording 1s very similar to the correct response as well as the 

/ 

stem. The last alternative is an example of an intuition distraction, 

/ 

since its selection is based on the "common sense" reasoning that since all 

magnets are generally a given shape, then shape determines magentism. 

An example from the same topic- area of an item at the middle level of 

comprehension is the following: 

Is it true that a magnetic compass which works" 
on Earth would jiot work on Mars? ' 

a. True -- because a magndt cannot- work 

in a vacuum. [Content distraction] 

b. True -- because Mars is too far from 

the North and South poles. [Intuition distraction] . 

c. False — because all bodies of matter 

have a magnetic field. [Logical fallacy] 

d. False — because Mars has a magnetic 

field of its own. [Correct] 
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This problem requires the students to know about magnetism in the sense 
that they must recognize some parameters under which that natural force is 
functional. Oistractors have been selected such that they test three 
several different error types of interest. The first detractor tests for 
content errors, since there 1s.no real reason for it being selected except 
that with the mention of vacuums it sounds scientific (the presence or 
absence of a vacuum has no elllct on magnetism). The second tests for 
intuition errors, since experience suggests the compass points north due to 
the presence of Earth's north pole, or that the focus of all magnetic 
fields 1s the polar region. The logical fallacy dlvvractor 1s based on the 
assumption that all bodies of matter must have magnetic field since the 

Earth does. ' ' 

An example of an item written to test the high level of comprehension 

1s Illustrated below. 

If we were sitting 1n an army tank, 1s 1t true 

that a compass will still function properly \ 
and point to the north? 



True — Earth's magnetic field 1s 

so strong. [Intuition distraction] 

b. True — since the tank 1s made mainly 

of steel. t [Logical fallacy] 

c. False — surrounded by steel, the 

compass will fall. [Correct] 

d. False — sound waves from the cannon 

will Impair the compass's 

operation. , [Content distraction] 

The problem requires the student to know how magnets work, what 1s or 1s 
not magnetic, and that compasses function as r. - H of magnetism. 

Clearly, several Items of each level are needed 1n order to accurately 
diagnose a student's level of comprehension. Also, each Item needs to 
cover as many as possible of the error types of Interest 1n order to ; 
r D ? r maximize the Information gleaned for diagnostic purposes. 



Experiment I 



Method 

Fifty three undergraduates in introductory biology classes at UCLA 
parti :ed voluntarily as subjects in their regular classrooms. Nine 
items were written for each area of science to be tested (Photosynthesis, 
Magnetism, Energy 1n Closed Systems, and Gravity) representing three Items 
at each level of comprehension, providing three opportunities to commit 
each error type. To these 36 Items were added four more items in other 
areas of science. Test administration was handled as a conventional ad 
seriatim paper and pencil-; test; ffi 
Results and Discussion 

Thefocus of the analyses conducted was to ascertain whether useful 

v 

diagnostic Information, emerged from the use of either levels of 
comprehension or error-type- o both. Protocols were scored for type of 
response made to each Item — correct, logical fallacy, Intuition 
distraction, and content distraction. Table 1 presents the mean • 
performance by level for each response type. 

The results were encouraging. First, 1f students failed at any lower 
level, they were unable to succeed consistently at a high level of 
comprehension. .For 89% of subjects injhe sample, missing more than one 
Item at any level of comprehension was accompanied by missing two or more 
items at the next higher level ( x 2 » U4.65, p < .001). When considering 
each content area in Isolate o the structure of the levels was maintained, 
although not to as great a degree. .On average, 66% of the students 
responding 1n each content area failed at all levels higher than an Initial 



error ( x^= 47.60; p < .001). This confirmed the expectation that the 
levels of comprehension represent distinct levels -of difficulty or 
complexity. The significance of this kind of information from a diagnostic 
test is that instructors could effectively gear their teaching to the 
precise level of comprehension at which a group, subgroup, or individual is 
presently operating. 

Second, we examined the protocols for evidence of consistent error 
patterns within persons. We wanted to know whether the distinctions, 
between error type would yield consistent diagnostic Information concerning 
the type of errors which a given person was committing. Analysis of the 
protocols for the entire test showed no discernible pattern of error 
consistency. However, when the topic arias were examined separately, three 
dlstir -t- patterns emerged. First, content errors were the most 
consistent. On average, six persons were Identified as making content 
errors within each area, a proportion significantly less than chance (x = 
5.29; p < .05)/. These were most probably guessing errors; people who 
guessed did so consistently. Their guesses were .based on "test-wise" 
strategies 1n which scientific sounding, or stem-Hke responses were 

selected, 5 

On "the average, 21 subjects per content area consistently committed 
intuition or logical fallacy errors( x 2 = 4.43; p < ,05). Eighty four 
peicc. of these subjects committed one or the other type of errors 
consistently at all levels of comprehension subsequent to the first such 
commission ( x 2 = 35.52; p < .01). Also, 1f either error type was 
committed at a low level of comprehension then one of the two errors was 
twice as likely to occur at the next higher level. The only problem 
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encountered with the intuition and logical errors was that for 38% of 
subjects there was a tendency to shift from intuitive to logical fallacy 
errors, or vice versa, hence clouding the precise diagnosis of a certain 
error pattern. This problem was especially evident in the areas of 
magnetism and gravity. However, this proportion was not significant (x^ = 
2.271; p > .05), and so we expect that in future studies, with more items 
and subjects, such a problem will not prove detrimental. 
Experiment II * 

A second study was conducted to investigate the degree to which the 
findings of Experiment I would generalize to other populations. 
Specifically, the questions of effectiveness concerning levels of 
comprehension and error types were addressed by administering a shortened 
version of the test to younger students of varied ability levels. 
Method V \ 

Seventy six students from a private junior high school for the 
"gifted" (IQ measured above 145)\in the west Los Angeles area participated, 
representing 38 seventh graders and 38 ninth graders. Sixty eight 



ninth-graders from a public junio^ high school in northeast Los Angeles 
also participated, representing three classrooms in which students were 
grouped according to common tracking procedures based on achievement. The 
latter subjects included 22 high, 20 middle, and 26 low achievers. 

Subjects completed a short version of the same diagnostic science 
test developed for Experiment I. Twenty items with satisfactory 
psychometric properties were selected from among the 40 items from 
Experiment I; in each^content area, two items were drawn from the high 
comprehension level, two from the middle level and one from the low level. 

• \ 
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Results and Discussio n 

Two sets of analyses were conducted, one to assess consistency across 
level, and one to asses consistency across error type. Table 2 shows the 
mean performance in this test by subgrouplng. The first analysis 
Investigated the levels of comprehension Ibedded 1n the Items. For the 
20-1tem test as a whole, results varied 1n consistency across the 
comprehension levels. Among higher ability students, the levels were 
relatively distinct and consistent; failures at a lower level of 
comprehension were accompanied by failures at higher level of comprehension 
for 67% of cases ( x 2 = 9.11; p < .01). Among students with lower 
overalls ability, the pattern was less clear/with reversals or sequence 
violations occurlng in some 47% of cases (x 2 * 20.28; p < .01), although 
only 16% were cases 1n which total failure at lower levels was associated 
with total success at the higher level; ( x 2 = 3.44; p > .05). The reasons 
for these contradictory results remain to be Investigated fully. Several 
factors might explain the contradictions, Including the reduction 1n number 
of test Items, differences 1n mental ability or maturity, and differences 
1n scholarship or classroom prepareness for test topics. 

The next series of analyses concerned error patterns. While 45% of 
private school students committed no errors at more than one level, 
generally . >e 0 levc. ( x 2 = 15.82; p < .01), 57% of the remaining 
students committed errors at multiple levels with consistent patterns for 
error type (xN 40.42; p.< .01). Unlike the patterns found for the 
college student sample, however, the most consistent patterns for the 
private school students emerged for logical fallacy errors. Public school 
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students products even less i ir-mt results. Error types were -ommitted 
in*a n«sarly random manner for all but 34% of the students, with none 01 ne 



Explanations for the lack of results in error patterns among both the 
public and the private school students center on two factors: the contrast 
in mental preparation with the college level group, and the reduced size of 



probability of systematic responses where one^misconceptlon might be 
reflected in several Item responses. Because of the reduced test size, 
students had fewer opportunities to respond to mips designed to re-measure 
the same concept and error types, hence less possibility for measuring 
consistent patterns. 
Conclusions 

The purpose of this study was to test the effectiveness of a prototype 
diagnostic science test. The test was designed to yield Information 
concerning two dimensions: (1) the level at which students comprehend 
concepts within certain areas of science, and (2) the specific types of 
errors in reasoning which they systematically commit. He find the results 
encouraging in that they show the possibility of accurately measuring these 
two dimensions of reasoning (comprehension level and type of error) for 
purposes which should be of high utilitarian value to instructor's. 



three public school groups showing a significant proportion of subjects 
producing consistent error patterns when data were examined by topic area 
( x 2 = 3.47, n.s.). \ 
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Traditional tests which yield only a total and subscale test scores provide 

petitioners with less information which is readily employable for 
remedial purposes. Th* promi. ? gnostic tests is that one can 
accurately assess not only the number of correct responses to. test 
questions, but also the precise type of errors which are being cojnmitted 
and the level of comprehension at which a student could be characterized. 

j ■■ 

The results of the science reasoning test discussed in toils' paper 

/ 

suggest a variety of future research avenues in the area of j diagnostic 
testing, studies which merit attention because of the valu^ of diagnostic 
information. The inherent utility of adaptively structuring the student's 
path through a testing session is yet to be explored fully. This would 
allow the student to guided directly from item to item In/a manner which 
matches subsequent items with patterns of abstractor response seen in 
preceding items. Computerization of the science reasonipg test is not 
difficult; the diagnostic interpretation available following an adaptive 
testing strategy could be enhanced relative to the conventional testing 

j 

administered In the present study. j 

The expansion of suitable item pools in which item distractors follow 
a logical order and item complexity 1s layered seems essential. Further 
work in diagnostic evaluation of science understanding would entail writing 
item distractors which adhere to the remaining five error factors of 
Showalter (1974). However, not every item can be or! need be accompanied by 
a distractor from . every error factor. Such^ expansion should balance the 
additional error types across all item complexities. There will be a^ 
mandatory increase in test length, but all possible, permutations of errof^ 
types with one another cannot be brought together without excessive numbers 
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of items. Thus further research in optimizing adaptive diagnostic 
test strategies is required. 
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Table 1 



Probabilities by Response Type and Hem Complexity 

Experiment I 



Item 
Compl exi ty 



High 
Medi urn 
Low 



Correct 



.58 
.78 
.93 



Log i cal 
Fal 1 acy 



.23 
.11 
.03 



Response Type 



Intuition 
Distraction 



Content 
Di straction 



.17 / 

.08 

.02 



02 
.03 
.02 
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" ' Table. 2 

Probabilities by Response Type and Item Complexity 

Experiment II 



I tern 
Complexity:, 



Group: 



Correct 



Response Type 



Logical 
Fallacy 



Intuition 
Distraction 



Content 
Distraction' 



High 


rUDNC LUn 


• 

10 


.10 


.12 


.12 




Public Medium 


.36 


.27 


-732- 


.05 




Public High 


.25 


' ' ■ .35 


.13 


.27 




D * i u a f a 7 

rnvalfi / 


37 

t J f 


.19 


.35 


.09 




Private y 


48 


.13 


.32 


.07 


pied 1 urn 


n 1 1 k 1 \ /* 1 All 

rUDMC LOW 


45 

c 


.27 


.21 


.07 




Public Medium , 


• D / 


17 


.17 


.10 




Public High 




: .22 


.22 


.08 




Private 7 


.58 


.24 


•11 


.07 




Private 9 

* 


.69 


.15 


.13 


.03 


Low 


♦ 

Public Low 


.66 


' .26 V 


.33 


.23 




Public Medium 


.27 


.12 


.09 


.08 




Public High 


.80 


.05 


.ii.: ' 


.04s 




Private 7 


.80 


.06 


1.12 


* .02 


PRir 


Private 9 


.90 


.02 


.06 


.02 
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ABSTRACT 



The role testing can play in ascertaining and improving the 
effects of educational programs and services is analyzed. Our point 
of view maintains that the connection between tests and instruction is 
best made integrally through an understanding of the design of 
learning tasks rather than through the use of techniques that attempt 
to join or to link the now-separate domains of instruction and 
testing. The context for task structures is described, and their use 
in developing instruction and tests is considered. The limitations of 
such an approach in practice are ..discussed and feasible approximations 
outlined. Finally, the research agenda in this area is broadly 
. sketched. . 
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INTRODUCTION 



The purpose of this article is to analyze the role testing can. 
play in ascertaining and improving the effects of educational programs 
and services. Our point of view maintains that the connection between 
tests and Instruction 'is best made integrally through an understanding 
of the design of learning tasks rather than through the use of 
techniques that attempt to' join or to link the ' now separate domains of 
instruction and testing. 

The focus on the design requirements of learning tasks represents 
a fundamentally different perspective on the test/Instruction issue. 
This perspective Is theoretically grounded In Its orientation deriving 
from research in learning, instruction, and .cognitive processing, to 
name but a few areas; yet, it also has numerous potential implications 
for practice. The context for task structures will be described, and 
their use in developing instruction and tests will be considered. The 
limitations of sich an approach in practice will be discussed and 
feasible approximations 'outlined. Finally, the research agenda in 
this area* will be broadly sketched. 

The Dimensions of. the Problem 

Public concern about the effectiveness of schools has led to a 
reliance on testing and test, results that 1s unprecedented in recent 
educational history (Airasian & Madeus, this issue; Haertel a Calfee, 
this issue). Tests play prominent roles in certifying competency for 
high school graduation, in college admissions procedures, and in 
conveying through the -publication of test results, the effectiveness 



of school district policies. These examples illustrate the practical 
and symbolic uses of tests. Test results are regarded as the "bottom 
line", and educators have devoted much attention to efforts to affect 
such scores, and thus graduate more students, place more in better 
colleges, and rank their district higher in test scores among other 
local school districts. 

Si nee. educators have accepted the validity of 1 tests as outcome 
measures, they have fed the public's desire for accountability through 
testing, and have created a demon that needs continually to be 
satisfied. Yet, the goal of Improving test scores is made extremely 
difficult by the ways in which schools ate organized and staffed, by 
constraints on their resources, and by the trends 1n the society of 
* which schools are onl-y a part (Bank & Williams, 1982; Zucker, 1982). 

Even 1f these factors were optimal, the problem of creating tests 
that are sensitive to the results of educational programs Is a 
difficult proposition. Part of the difficulty stems from the way 1n 
which testing, as an enterprise and a research area, has developed and 
part from the existing scientific, conceptual framework of testing. 
I-n our view this framework 1s different from the conceptual framework 
of Instructional design. 

Testing 1n the Conceptual Framework of Science 

* A scientific and experimental orientation appears paramount 1n 
the psychometric view of testing. The practical uses of testing (to 
create a record of student performance) are subordinated in favor of 
the scientific use of ; tests (to detect individual differences or to 
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determine effects of interventions). The focus on the latter purpose 
has led to a preference for the design and development of tests that 
best support these uses. For instance, in the Beginning Teacher 
Evaluation Study (BTES), items were not selected for inclusion on the 

dependent measure based on their fidelity to Intended learning and 

r 

Instruction, but rather for their correlation in pilot data with the 
independent variables of interest (Filby and Dishaw, 1975). This 
procedure increases the probability that the test will find what the 
researchers are looking for, but does not Insure that learning 1s 
assessed adequately— because the nature and definition of learning 
never emerges as a primary Issue. 

Test development from this scientific view follows a well- 
established, scholarly process. Tests are developed to assess a 
construct. Items for the test* are selected or created, and data are 
collected on performance. . Items are Included, dropped,. or revised 1n~ 
accordance with the fit the Item data provide to the posited model. 
In this orientation, the psychometrl dan's job 1s to hypothesize a 
construct or trait that 1s assumed to be measured by a particular Item 
set, and then to use that set to observe nature, to report reliably 
what exists, and to revise the test or to reformulate the construct to 
explain empirical facts, processes, and their relationships. The 
sc1ent1st-psychometr1c1an's mind 1s active, but since his/her role 1s 
a descriptive one, that 1s, portraying how students respond to sets of 
Items, the scientist remains outside of the action of Instruction and 
passive with respect to creating "better" performance. The focus 1s 
on accuracy rather than on Improvement. In fact, among the most 
serious errors a scientist can make is to perpetrate reactivity, where 
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Inadvertent effects are produced by the process of measurement 
Itself. The measurer should make no ripples. This is one prong of 
passivity 1n world view. 

A second prong of passivity derives from the notion of stability, 
a central thesis underlying much of science and measurement theory. 
The constructs to be measured are treated as stable and are described 
as traits or constellations of responses thought to persist 1n time. 
Classical psychometric and statistical theories have developed under 
the assumption that stability, regularity, and predictability are at 
the heart of scientific Inquiries. A concommltant assumption, espe- 
cially pertinent to this discussion, 1s that measured individual dif- 
ferences probably endure over Intervention. Evidence for this point 
of view can be found in the literature on measuring change, where 
growth and measurement error are sometimes treated almost Interchange- 
ably (Harris, 1963). • That the early uses of tests were for placement 
1s no surprise, since labeling and grouping Individuals 1n homogeneous 
clusters. IsTa logical outgrowth of the belief 1n stability. 

The conflict 1s most simply that education 1s an enterprise 
directed at producing change, yet our tests and the psychometric 

and description. Traditionally, measurement's role has been that of 
the objective, outside observer who analyzes Its subject from afar. 
The serious Integration of test1ng-*n<k1nstruct1on and the use of 
tests for Instructional Improvement, hoiever, require a more active 
perspective that Incorporates an 1ns1d^ view of the phenomena qf 
Interest, learning and Instruction 1n the- educational process. . 

SO 
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Conceptual Framework for Design 

As an alternative to the scientific perspective adopted by 
psychometricians\ design methodologies reflect a different point of 
view. Design 1s a process that synthesizes practical and theoreti- 
cally-grounded Ideas to produce a procedure or product that changes 
the environment; it is an explicit problem-solving activity that 
generally Includes notions of planning, creation, and fine-tuning. We 

» 

are all familiar with design methodologies, from the most obvious 
aesthetic application (graphics, Interior design), those which blend 
aesthetic and technical features (architecture), to those which 
emphasize the applications of scientific findings to particular 
problems (computer design, engineering of all sorts, medicine). 

Most professional schools are committed to training at least some 

> 

of their graduates 1n the design (as opposed to research) paradigm, 
although 1n education, as 1n other fields, design has somewhat less J_ 
status than research activities. Because design creates things that 
must operate 1n reality rather than contend solely with the elegance 
of Ideas, Its place 1n the academic community is tenuous and probably 
— survives ,bACJL^e_j5M!e_deslgnjractitj.oners ( doctors and 1 awyers ) are 
perceived to be worth high levels of financial reward. Additionally, 
although the outputs of design may Include scientifically-based 
processes, 1t 1s undeniable that art or craft 1s also demanded^ Thus, 
design work gets labelled /'atheoretlcal ," and Its status denigrated. 
Even status. aside, because we know less well how to help people 
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become talented designers than competent researchers, de>1gn aspects 
1n education are often neglected. 

In contrast to the researcher, the task of the designer is not 
descriptive. Rather, from the outset, the designer's task is to 
Improve upon present practice. Teachers 1n our public schools are 
design practitioners, however Informally they accept that role. 



Their task 1s to combine Informat 



on from a variety of theoretical 



approaches with practical wisdom to affect the quality of education. 
Change is the goal, of a designer and thus that orientation would seem 
to serve education well, and particularly the problem of connecting 
tests and Instruction. 

Even 1f one were to minimize the effects of these two different 
frameworks on the predisposition for action or reflection, cr perhaps 
admit that there 1s a continuum rather than two mutually ^xcl usive 
perspectives, the serious problems remain 1n connecting testing to 
Instruction and promoting school-based change. These problems Inhere 

1n the realities of school operations and 1n the nature of effective 

Instruction. 

- - -■- - Context: The Worl drof ~ School s 

Because psychometrics 1s a scholarly pursuit, most of Its work 1s 
conducted 1n settings remote .from current public school experience. 
Psychometric researchers don't often go to schools, and when they do, 
they usually don't focus on Instructional Issues. Because of this 
lack of familiarity with the lives of people 1n schools, 1t 1s not 
surprising that some misperceptlons seem to have occurred. 
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One, set of misperceptions involves teachers and what it means to 
teach something. Another concern is the extent to which the process 
of curriculum development, adoption, and Implementation occurs and can 
be counted upon to provide a common context for school events in 
different schools (Sirotnik, 1981). For Instance, the irrationality 
of the system as it most frequently operates causes only an occasional 
lament on the part of psychometricians. Consider that school curri- 
cula, and therefore, most of formalized content, are developed outside 
of the schools and marketed by text publishers. The. same Is true for 
tests used in schools. Unfortunately, the coordination be'tween test 
and curriculum development- is nil. Thus, at the mo S t.bas1c level , we 
can show that content differs in tests and in texts (See Floden, 
Porter, Schmidt & Freeman 1980) and that at the grossest exposure 
level,, students cannot be expected to, perform with regard to content 
they have not seen. A recent" study comparing district curriculum ' 
bbjectlves in both language and mathematics with the state assessment 
and various standardized tests Illustrates this Irrationality. - Some 
tests Included only 25% of the district curriculum, and, 1n some, 
almost 50% of the test was not covered by the curriculum (Cabello, 

1982). I J 

" ; But attention to currlcular match demonstrates only very global 

concern for the relationship betwe'en testing and Instruction. Such a 

concern assumes that formal curriculum 1s, 1n fact, Implemented and 

that simple exposure to content 1s sufficient for students to learn; 

both assumptions are unfounded. In seriously coordinating testing and 

instruction, one Immediately 1s confronted with the complexity of 

actual instruction, Including, for Instance, how learning takes place, 

93 
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the range of extent presented, the context in, which a set of skills 
should apply, and how teachers augment or circumvent existing text 
material to facilitate learning. One might be tempted to ignore these 
confounded variables and to try to bring order to the system gradual- 
ly by focusing on and controlling only one set of variables at a 
time. Yet, when one focuses on one aspect, say, content, and attempts 
to match tests and curricula on that basis, one necessarily Ignores 
other Important considerations that ; contribute to the Irrationality of 
the present system. The status quo, as a result, is Inadvertently 
perpetuated . Unless integrated alternatives are pursued 
deliberately, the status quo with Its irrational base will continue to 

be our only option. 

We propose, therefore, to use design methodology to create an 
improved system for education and learning. This approach concen- 
trates not on Unking extant curricula with extant tests, but Instead 
on the complete design of an entire system; In Vac t, a redesign, so 
that the entire system makes sense. Instead of studying testing 
properties In terms of existing rules of order, we propose to focus on 
the learning tasks of students. The system starts with the nature and 

definition of what is' to be learned-the task structure; the charac- 

' - 1 

terlstics of Instruction and of testing then follow naturally and* 
rationally. Our proposition 1s that by designing task structures; we 
provide a model of the features that learning-should exhibit. The 
model causes the requirements for testing and teaching to converge 
since they share, by definition, critical features. Linkage becomes 
redundant. 



. r 94 



ERIC 



Task Structures 



The cask structure approach is based upon the design of the 
learning tasks desired of the learner. This structure is specified by 
a series of mles or examples; the final object 1s to present as 
clearly as possible the expected set of skills with regard to specific 
content. Task structures integrate critical ideas 1n education by 
reserving a place for them in the structure Itself. For example, the 
range of content over which the learner's skill 1s to generalize, the 

manner 1n which transfer is treated, the behavioral formats for 

exhibiting performance, the level of cognitive operations required, 
and the complexity of language are all explicitly treated 1n the task 
structure. f 

He will treat the features of a task structure 1n turn and 
present definitions and descriptions, theoretical connections, and 
examples. 
Task Description 

The first element in a task structure 1s the general description 
of the task. This statement may be thought as equivalent to the 
statement of objective 1n objective-referenced tests or outcome 

statements used to guide the development of criterion-referenced * 

- x \ 

tests. Its purpose Is simply to direct and drcumscr1beXttent1on to 

\ \ 

a general area of content, such as geometric proofs, and to focus on 
the type of skill needed, such as to solve problems or to demonstrate 
procedures. Such statements often have served as the sole descriptor 
for developing criterion-referenced tests. However, In a task 
structure, the statement serves principally as a convenience, as a way 
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to get Into or to approach the more taxing endeavor of describing the 

& 

learning requirements of a particular task. 
Content Limits 

Establishing the content limit of the task is an initial prob- 
lem. These limits are intended to circumscribe clearly the substance 
or content upon which the learner is supposed to operate. WMle the 
task description describes content in very general terms, content 
limits make more specific the, particular elements of content to be 
included, and thereby help, to make explicit requirement for instruc- 
tional exposure arid opportunity to learn. Two common approaches to 
content limits have been advanced: definition by curriculum and 
definition by agreement. 

The first approach responds to. the real world of school exigen- 
cies and suggests that content be defined by reference to extant 
curricular material {Baker, 1974), such as specifying permissible test 
content for a reading comprehension objective to include non-fiction 
selections occurring in a particular 9th grade district adopted 
— textbook-^Ifl-thlslnstance, the content is selected based simply upon 
the rule of potentiaV^tpasWe, and.while opportunity to learn is ah 
obvious criterion, the seleltion^of^substantlve material has .been left 
to the indeterminate judgment of textbook writers and publishers. 

Unfortunately, analyses have shown that systematic--,structure and 

\ •/ 

features that contribute explicitly to learning are/ often absent in 
~ commonly used texts (Quellmalz, Herman & Snidman,^1977; Herman, 
Hanelin & Cone, 1977). The reading selections at a particular grade 
level, for instance, do not appear to follow inferrable rules of 
progressive linguistic, semantic, or syntactic complexity. 

ERIC 1 
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A second common approach to content limits involves developing an 
agreed upon set of -boundaries which is disseminated to both teachers 
and test writers. The benefits of such agreement is that the proba- 
bility of a fit between instruction and testing is increased simply by 
^Xcommuni cation, permitting an instructionally-focused outcome system 
(Popham, 1981). ^The criticism of such an approach rests upon the 
arbitrariness of the content boundaries selected (Why four line 
paragraphs instead of five or three line in a reading comprehension 
.task, for example). This charge of arbitrariness has been countered 
by appeals to the wisdom of reliance on "human judgment. 11 

We suggest that arbitrariness of content limits may be mitigated 
through reliance upon relatively strong theoretical or empirical know- 
ledge about learning (as well as on human judgment) to decide what 
content limits are sensible. ' This approach substantially supplements 
the rationalization of content based upon probable exposure to extant 
curricula, e.g., a 9th grade textbook, and the human judgment defense, 
and permits each element of content limits to respond to potential 
issues in learning. We propose using two well documented constructs 
related to research on learning to define content limits: 1) general- 
ization and transfer; 2) quality of discrimination/performance. 

Generalization and Transfer . The concepts of generalization and 
transfer help alleviate the anathema "teaching to the test". Psycho- 
metricians, perhaps because of their frequent concentration on indivi- 
dual Items rather than on concretely related Item sets, seem to worry 
about this problem a good deal . Not acknowledged 1s the fact that 
"teaching to the test" can occur 1n two forms: teaching the exact 
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items that appear on the test and teaching the class of content, by 
sampling, that the test is designed to measure. The former appeals to 
rote learning and is usually of limited educational value; the latter 
demands that we have a solid notion of what learning is intended and 
that we focus on significant higher-level tasks. The content HmTtsr 
should create this notion of the class of substance to which the test 
behavior is meant to apply and generalize, echoing the general idea of 
domain-referenced testing (Hively, Patterson, & Page, 1968). 

One question here is to what topic or topics is the behavior 
supposed to generalize? The answer to that question depends generally 
on practical matters, such as the length of instructional time avail- 
able for the task, or more directly, to the level of specificity at 
which the task 1s to be learned, as well as interrelationships among 
potential topics. For Instance, a learning task related to conjugat- 
ing and applying "-ar" verbs 1n Spanish might be defined with the 
expectation that the learning should generalize to all such verbs. 
Alternatively, the task could be expanded to all regular "-ar, -er, 
and -ir" verbs, requiring a longer allocation of Instruction. The 
conjugation procedure for these Spanish verbs 1s essentially the same; 
only details change and change 1n predictable ways. Research on 
generalization and transfer would suggest that teaching students the 
critical features of such conjugation and providing related practice 
would be sufficient for them to apply the rules to any. new examples of 
regular Spanish verbs, and thus would support the more Inclusive task 
structure. In contrast, the coherence and research support would be 
more problemmatlc if the content for a task in Spanish Included verb 
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conjugations, pronoun number, and sentence expansion. The probability 
that transfer would occur is low, precisely because the information 
and concepts necessary for success vary. One decides on what level of 
generalization and transfer one can achieve partly by relying on 
unfettered human judgment, but partly based upon theoretical or 
empirical evidence about how knowledge in a particular, subject matter 
area is connected (Geeslin & Shavelson, 1975). 

Specifying the' topics over which performance is to generalize 
serves an additional purpose: if transfer between topics is not 
explicitly taught, there is little reason to bel ieve v transfer will 
occur (Silberman, 1964). The definition of task structures in written 
composition, for .example,, relates to the low transfer of writing 
ability between topics, such as "My Best Friend" and "Credit Card Use 
in the United States" (Quellmalz, Capell, & Chou, 1982).. Students 
need to learn that the same strategies and skills are applicable 1n 
both cases. The explication of the range of topics oyer which perfor- 
mance is expected or desired to generalize, when communicated to those 
responsible for Instruction, then, can Itself facilitate transfer. 

Since tests take considerable time from Instruction, they probab- 
ly should be perceived seriously and reserved for those goals that 
incorporate generalization and transfer rather than for memorization 
of specific content. Desired levels of generalization and transfer 
among topics should be specified based upon the level of effort 1n 
instruction and theories of content relationships derived "from 
analyses of the discipline (1n addition to the more common cur ricular 
and consensual bases described above). 



A second area to be considered under the category of generaliza- 
tion and transfer relates to the form in which information is presen- 
ted to the learner. Form is not item format, such as passage length 
or number of distractors, but rather the substantive features of the 
task, other than topic, over, which the learner is expected to general- 
ize. For example, "triangles" is one topic in the task of learning to 
discriminate confusable geometric figures. To ascertain success, the 
learner can be asked to discriminate the correct answer presented in a 
single form, e.g., your standard, equilateral triangle. However, if 
one wants to assure an understanding of a triangle that is somewhat 
more robust, one would provide students with correct answers that 
Include acute or right triangles, and perhaps triangles whose vertex 
is not* perpendicular to the margins of the page. Students might be 
asked to find the triangle when other salient perceptual cues, such as 
size and color, might interfere. Here the issue is clearly "over what 
cases does the learner recognize a triangle?" 

Form and type of information can be illustrated In non-perceptual 
concept learning as well. What class of information will the learner 
be expected to have acquired in order to attempt the task? For 
example, in written composition, a question may be posed about whether 
the learner has sufficient knowledge about a topic to write about 1t. 
How is that information to be provided? How complex will be the form 
in which the information is presented? Will it be a list that the 
- learner simply has to transpose into prose? Will he/she be expected 
to infer meaning from embedded and subordinated information? As a 
second example, consider the task of learning to -identify the main 
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idea of a prose passage, a common enough objective of reading instruc- 
tion. What type of passage will be presented? Will it be one in 
which there is a single clear main idea? Or one in which two partial- 
ly developed ideas compete with a third "main" idea for dominance? 
The differences in the task intellectually should be clear, and the 
different requirements for instruction are probably obvious. * 

We are recommending, then, that issues of generalization and 
transfer be incorporated in the content limits section of a task 
structure specifically to address the topics oyer which the response 
is supposed to generalize arid Information or presentation forms over 
which the response should "transfer. Both of these areas are to 
supplement the simple notion of opportunity to learn, defined either 
as content 1n required texts (Floden, fit al , 1980), or tirne on task, 
i.e., time nominally allocated to particular topic (Denham & 
Lieberman, 1980). It 1s our belief that these instances are too 
global to relate productively to, learn ihg tasks. In addition, we 
believe that attending to generalisation and transfer strengthens 
human judgment because theoretical and empirical bases are used for 
content, selection rather than more vague appeals to authority. 

D 1 scr 1 ml nat1 on/Fer^ A_second,^general_area 



within the content limits section of task structures focuses on the 
standard of performance expected.of the learner. It 1s at this point 
where claims of "educational excellence" are based by defining the 
required quality of response. However, performance quality should not 
be confused with common versions of performance standards (Mager, 
1961; Popham 4 Baker, 1968; Anderson & Faust, 1973), all generated 

i 

i Directly related to thevlssue of form of Information Is the 
linguistic features of tHe text of the test Items. However, we will 
separate that discussion Into a later component of task structure 
because the theory ,wh1& supports such analyses differs from the 
cognitive research base of the present section. 101 
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during the behavioral objectives era of the sixties and seventies, 
performance quality relates to non-quantitative features of responses $ 
that illustrate the level of refinement of the response. Because 
student response options fall into two major categories. Let us 
illustrate this principle in both selected and constructed responses. 

In a selected response task such as the triangle discrimination 
task described above, the a priori difficulty of the task depends upon 
^not only the range of correct answers the learner has to identify, 
e.g., isosceles and acute triangles, but the fineness of discrimina- 
tion required to make that Identification from detractors. Very j 
little refinement would "be required to select from detractors that 
consisted of those in panels a or b in Figure 1, Consider, however, 
if the detractors consisted of those shown in panel c. An analysis 
of these latter response options should demonstrate that the Item 
requires relatively fine discriminations and exhibits higher _a priori 
difficulty. 



Insert Figure 1 About Here 



The analysis also should clearly demonstrate that choice of 
distractor provides diagnostic Information about the class of mistake 
the. student 1s making. To select the first option 1., panel c, the 
student would have to believe that open as well as closed, three- 
sided, straight-lined figures met the definition of triangle. For the 
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(1) (2) (3) (*) 





Figure 1 

Alternative Distractor Options for a Triangle Discrimination Task 
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second option, the student would have neglected the three-sided aspect 
of the concept. In option three, the student would have overlooked 
the requirement of straight sides. In each of these instances the 
provision of diagnostic information is preplanned. 

In contrast, much of the extant literature in diagnostic testing 
(see Brown 4 Burton, 1978; Tatsuoka 4 Tatsuoka, 1980) lacks such a 
priori design of detractors to yield explicit diagnostic Information; 
instead,, sources of error are inferred from item response patterns. 
Explicitly including rules'for the creation of wrong answer domains in 
the content limits can significantly increase diagnostic power, and to 
the extent that such rules incorporate research on concept learning 
(such as Tiemann 4 Markle, 1973; Tennyson* Wooley, 4 Merrill, 1972), 
the diagnostic quality will be more refined. Where concept learning 
is not the focus, content limits for multiple-choice items may be 
generated specifically to deal with aspects of the task that may have 
been underl earned as well as those aspects that may have been mis- 
learned. •• '» 

In the case of constructed response, where no dlstractors are 
provided for the learner, the content Hmlts should account for the 
expl 1c1t standards tW^ ~ 
student effort. These standards, or criteria, are applied to student 
products to assess the extent to which products, such as essay answers 
to science questions, or English compositions, exhibit desired 
features! Such decisions can be reached through holistic approaches, 
where the overall value of the paper 1s judged by Internal standards; 
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or by analytic methods where particular aspects of student production, 
such as style, coherence or grammar, are separately considered. In 
either the holistic or analytic. approach, the response may be judged 
according to a check list (where the paper, or the style, is either , 
satisfactory or not), or through the application of a rating scale 
(where points from 6 to 1 depend upon the quality of student perfor- 

mance.) ' 

It should be clear to. see that less well explicated scoring 
systems, i.e., holistic, rely more on undifferentiated human judgment 
and experience, whereas explicated standards, such as analytic 
approaches with logically anchored rating scales, provide much more 
information about student performance. This additional information is 
desirable in task structures for it directly implies the type of 
instructional task/ the learner is expected to encounter as well as 
the remedies that may be necessary to address inadequate performance. 
Explicated standards for judging criterion responses, thus, are an 
important component for teaching and testing. 

With the specification of content limits, performance quality is 
measured by design, either inherent in the level of discrimination 
required 1n selected responses or by the explicit statement of criter- 
ia in production responses. Difficulty emerges directly from the task 
structure design and 1s a function of task complexity and fineness of 
required discrimination rather than created anprlcally by proportions 
of people who succeed at an item. This conceptual design of difficul- 
ty may help break the tautology that exists between empirical "Item 
difficulty" and assessment of the effects of Instruction. Such an 
approach also allows one, by' reviewing wrong answer choices, for 



instance, to determine when partial learning has occurred and where 
remediation is needed. „ 
Linguistic Features 

Linguistic features are another important aspect of task struc- 
tures, but their role in test and task design has been treateB in 
general IV disjointed fashion. Level of difficulty has been assessed 
by various readability formulae which take into account the difficulty 
level of words (inferred from developmental or frequency measures), 
and sometimes the complexity of syntax (Duffy, 1981). Yet more 
complex linguistic structures play a role in tasks that either present 
verbal material as stimuli, including verbally stated alternative 
responses, or include rating systems based on verbal products by the 
respondent (Duffy, Curran, ft Sass, 1982). Particularly when non-^ 
native English speakers are assessed, the variation In performance 
created by apparently casual linguistic options may be great. Bauman 
(1982), for example, found that problem types Identified through 
linguistic analysis posed serious difficulties for readers-problems 
that were not directly related to the construct being assessed. 

Systematic attention to the linguistic components of tasks may 
permit more accurate assessments of true performance levels. Measures 
of linguistic complexity need to be created that are appropriate for 
^bothJong and short verbal passages and which Include some notion of 
deviation from semantic and syntactic experiences of the respondents. 
For instance, some difference score may be obtained depending on the 
compatibility of the sentence patterns with the native language, or 
the root of more difficult words and the native language. To the 
extentUhat language proficiency 1s not an Inherent feature_,of^the~~ 
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task of interest, then effort should be made to purge verbal materials 

of unnecessary complexity. 

Cognitive Complexity 

Another critical feature of a task structure is the cognitive 

complexity of the task. Simply stated/cognitive complexity is the 

intellectual "level" apart from content,, at which the. learner is 
expected to perform; These levels have been taxonomized by Bloom and 
his associates (1956) to include six categories, Knowledge, Comprehen- 
sion, Application, Analysis, Synthesis, and Evaluation. Presumably, 
each of these categories refers to cognitive processes that are 
successively increasingly complex as well as dependent upon prior lev- 
els. A slightly different structure has been posited by Gagne (.1975 > 
where essentially stimulus-response learning, multiple discrimination, 
concept learning and problem solving form the major dimensions of 
intellectual skills. Simplifications of these schemes have been found 
in the cognitive literature (Quellmaiz, 1982) where principal .distinc- 
tions have been made for tasks whose purpose 1s the storage, associa- 
tion, and retrieval of information contrasted with tasks requiring 
processing of Information, Including subordination, reconfiguration, 

and other adaptive processes. 

The task structure must clearly provide an Indication of the 
Intended cognitive .complexity of the task. This ascription will 
relate to two features of the content limits already described^, 
generalization and required performance quality. First, complexity 1 
a function of the degree of expected generalization and transfer, jnd 

■-. . 

1n the nature of the required performance,' e.g. , the number of cues 
... provided, the amount of Information that must be subordinated should 
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be reflected in this area. Second, complexity is also a function of 
the performance quality that is demanded. A good example might be a 
problem solving task involving the correction of a operating defect on 
a jet aircraft. Perhaps no transfer is necessary, for it is the F-14 
and x only the F-14 that is of interest. However, because of the 
enormous inherent complexity of the circuitry of various systems, the t 
"task requires within it a high level of discrimination and therefore, 
has high cognitive complexity. 
Format 
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Another related feature of the task is the format in which the 
assessment is made. Format includes both the descriptive mode 1n 
which the task\is presented, e.g., print, graphics, video recording, 
and the form in which the response is desired, e.g., multiple choice 
with four response options, written composition, and so on. Obviously 
the format relates both to the practical matter of presenting and 
obtaining task related Information as well as to the requirement to 
incorporate specifications of task structure identified in content 
limits, in linguistic features, and 1n cognitive complexity. It is 
possible, for Instance, that format 1s truly an unimportant issue, and 
that fact 1s demonstrated by the expectation that students will be 
able* to demonstrate task mastery 1n one of many formats or in all of a 
number of formats. 

The extent to which format dependence has taken over from o opt1mal 
learning requirements of tasks 1s documented by the attention the 
general education system directs to test wlseness. Here the format of 
the test 1s regarded as separate from, and sometimes equal to (1n 
Importance), the content and Intellectual skill demanded by the task 



itself. Including, format as a particular dimension of the task struc- 
ture allows for the rational review of the role of format and its 
relative importance or subordination to issues of content mastery. In 
addition, the Internal consi stency of /descriptions about generaliza- 
tion of content performance quality and cognitive compl exity can be 
assessed in reviewing the format! s)' projected for task demonstration. 

'/' r. 0 

..Instructional implications of Task Structure Dimensions 

• - J 

The premise of this paper is that attention to task structure 
dimensions outlined above provides a common focus for and defines the 
•structure of assessment and instructional systems. In this section, 
we propose to identify the aspects of the task structure that inexor- 
ably iead to instructional decisions. The problem in relating 
Instruction and assessment changes dramatically. Instead of dealing 
with the amount and degree of overlap between activities and arti- 
facts, one focuses, on the degree of Implementation of the task 
structure Itself, la far different task Intellectually, and with the 
potential, at least, for greater satisfaction. 

In dealing with Instruction, let us exclude from our discussion 
issues related to affective, motivational or social learning paradigms 
and focus, for purposes of our analyses, "on the cognitive and behav- 
ioral tasks of learning and teaching. Clearly, based on the litera- 
ture 1n instruction (Bower & Hilgard, 1982; Gagne. 4 Briggs, 1981;.^ 
Traub, 1966), a critical Issue is jjp extent to which students have ^ 
been exposed to a particular task and in fact have had the chance 



to practice it: under conditions implied by the content levels, i.e., 
with both particular and generalized examples, and at the level of 
performance quality (such as discrimination) , impl ied by the task. 
However, opportunity to practice criterion behavior is necessary but 
may not be sufficient for less able students, and more complex tasks. 

If criterion behavior is too complex to be acquired by repeated 
rehearsals, what-Should be done first, what. component skills must be 
acquired? Unlike many statements of objectives, the subordinate ... 
. components of instructional tasks are Inherent In the task structure 
•itself. The identification of features over which the performance 1s- 
expected to transfer specifies a set of experiences for the student. 
For example, if the task structure 1s to be able to analyze particular 
propaganda devices 1n advertisements, news articles, editorial s, and 
verbal appeals, then students would need practice with all specified 
media as well as instruction and practice with each specified device. 
The Indicated embeddedness and subtlety of propaganda use would > 
similarly suggest the successive range of difficulty that would be 
appropriate for instruction. In other words, inherent 1n the task 
structure 1s a plan for successive approximation of the end desired 
learning tasks, where Individual components are practiced and then 
combined >1n Increasingly complex sets.;, 

The nature- of Instructional tasks also follows from the specifi- 
cation of content limits for performance- quality: the classes of 
concepts Included 1n the detractors, or the criteria by* which the 
ultimate student product 1s to be judged. For Instance, 1n the 
triangle discrimination task described above, Instruction would need 
to take clear and differentiated account of the attributes of the 



triangle of interest: it is a geometric figure; it has straight 
lines; it is closed; it is three sided. The order in which these are 
treated or the motivational context in which these attributes are 
introduced make little difference to this analysis. The implementa- 
tion issue is the extent and degree to which these attributes are 
treated, i.e. the extent to w hich instruction_ afl<t practlfifi-deal-wltb— ^ 
each attribute, singly and/or in combination, which represents a 
significantly more refined view of opportunity to learn. 

Similarly, in constructed responses, if a learner's writing is to 
be judged on his/her use of coherent sentences in a paragraph and the 
choice of development used in the paragraph, then the instruction 
must, in a differentiated way, treat these options. Again, the 
context in which instruction occurs or the instructional approach, 1s 
not of first concern; matters of presentation style, sequence, etc., 
are not the primary focus because valid differences cannot be dis- 
cerned in the absence of specified treatments, treatments which are 
directly relevant to and derived from the desired learning. Once 
more, the Issue becomes whether the elements of the task structure can 
be found in the Instructional provisions for the students. It 1s an 
Implementation problem, looking at frequency and Intensity, rather 
than a problem of determining overlap. 

Metastructures for approaching these Individual instructional 
components of the task structure depend on the educational philosophy 
and instructional style preference of the teacher. Direct Instruction 
(Roserishine, 1982) and task analyses (Gagne, 1977) approaches to 
teaching may be appropriate. On the other hand, a less directive, 
mo*rf 1nqu1ry-or1ented approach may be preferred. Most Important to 
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note, however, is that the action changes from attention to the 
process of instruction, or how instruction occurs, to the substance of 
instruction and the modelling of the structure of task itself. 2 Our 
belief is that the way to outcomes is far easier and of secondary 
importan ce if the quality of out come des ired is sophisti ca ted and well 
described. Targetted instruction under whatever approach, will likely, 
be more effective than more diffuse attempts. You have to teach "it" 

I 

if "1t" is going to be learned. 

Applications of Task Structures 

Since 1t 1s obvious that the rhetoric of design and change 1s 
insufficient Itself to create the conditions for implementation 1n 
education, what 1s the likelihood that such an approach 1s practical 
at all? Organizations responsible for Implementing educational 
practices like public schools, are often not change-oriented them- 
selves. They woul d rather adopt the surf ace appearance of change and 
Innovation (Plncus, 1975) than- to undergo the dislocation that real 
change Implies. 

Having laid out our Ideas on task structures and the promise they 
hold for making the educational process more rational , fair, and 
Instructional^ effective, let us consider their possibilities 1n 
practice. Or have we, Hke the academic friends we've criticized, 
proposed an Ivory tower system that will not survive the test of 
reality? 

First, let us consider a serious distortion of the Ideas we 
espouse: minimum competency testing. Essential In this movement 1s 

i t 112 
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the idea that schools should be responsible for assuring the acquisi- 
tion of particular skills—learning tasks— and that these skills 
should be the subject of both instruction and testing. Yet, in 
practice, the target skills do not truly reflect school and teachers' 
main goals, and the natural linkage of instruction and testing within 
"ThTsystem has~noT" often occomrdr— Insufftd«ht -technical-expertise, — 
often volatile political environments, and high stakes have combined 
to produce more rather than less irrationality: ninth grade— or 
lower— skills many represent essentially a riew one-year remedial 
curriculum masquerading as the minimum competency for high school 
graduation. Such may be the fate of most top-down change mandates 
that attempt to solve complex educational problems with simplistic 
solutions that are insensitive to local context. 

Our experience, however, Indicates that more positive outcomes 
are possible, and that approximations of our learning task approach 
are feasible 1n practice. Below we allude to two approaches we have 
used to implement task structures. The two examples vary 1n the local 
motivation for change and the source of educational goals— or the 
learning tasks to be accomplished. The examples illustrate a "mini- 
mal" and "maximum" attempt at change. 

With a minimalist view, one school district attempted to solve a 
common district problem, "Raise those test scores." Learning tasks 
were directly Inferred from the actual content of jthe_tests-1n ques- 
tion, I.e. the task structures were deflnelf to parallel state assess- 
ment test content.- District curricula were analyzed to determine the 
extent to which instruction and practice were provided for each 
learning task. Little direct and explicit overlap was found and 
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supplementary practice exercises and cues for instruction were devel- 
oped to fill in the gaps. Additionally, test performance was analyzed 
school -by-school within the district and school specific instructional 
prescriptions were created. School -wide strategies and explicit,/ 
Instructional guidance and materials for teachers and students were 
designed. The entire effor t was_1n1t1ated centrally and received 
strong district leadership and subsequent principal support. While 
some might question the validity and value of such "teaching to test" 
activities, the effort was directed at Instructional" Improvement, 
based on the goals measured by the test, and served, the practical 
needs of the subject school district. 

More comprehensive change efforts have been conducted ;1n other 
local contexts, using a more grass roots approach. Several change 
efforts conducted by the UCLA Center for the Study of Evaluation have 
used a multlprong curriculum-assessment-staff development strategy. 
In these Instances, teachers have been trained 1n the task-structure 
approach to Integrating Instruction and testing and 1n sound test__— -- 
development techniques. Teachers then playjthe^actlve" role, w1|h some 
technical assistance, In, defining critical learning tasks for their 
subject area, 1n explicating the dimensions of each task and 1n 
constructing suitable test Items. The resultant tests are subsequent- 
ly used to diagnose Individual , class, school, and district needs, and. 
to monitor student achievement. Model Instructional approaches and 
teaching lessons for the target learning tasks also support the 
f . process. r 

More comprehensive Implementation of task structures 1s 
possible. Applications 1n emerging technology and 1n the private 
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sector in highly technical training environments represent two poten- 
tial opportunities. In both cases, the incentive for high quality 
training may be possible to an extent not present in public school 
education. In the second case, there are controls on the selection of 
the group to receive education, either because they are hired or 
otherwise screened, and teaching conditions and student motivation are 
more tractable. The use of technology Is a seductive arena not only 
because the personal and idiosyncratic mediation of instruction by 
teachers will be avoided, but also because of the possibilities for 
closer monitoring and Immediate feedback with refined branching and 
remediation options. 

. Theoretical and Applied Researchjssues_^ 

The fole~of~theory. 1n research on task structures 1s obvious. 
However, the theory of Interest 1s not psychometric theory, but rather 
propositions that grow from perspectives In cognitive and behavioral, 
learning 1n the field of psychology, 1n psycho! 1ngu1 sties, and In 
contrastlve linguistics. A practical Issue relates, once more, to the 
level of generality necessary and the Inherent relationships among 
features of the task structures. For example^ can one have relatively 
simple content and require sophisticated cognitive processes? The 
answer on a single Instance level 1s "of course", but how general 1s 
that answer?. What 1s the relationship between language complexity, 
cognitive processes, and transfer and generalization of content? How 
circumscribed or broad can a task structure be; that 1s, what are the 
limits or optimal levels of generalization?. These and other more v 
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provocative questions need exploration as well as testing in alterna- 
tive contexts and degrees of implementation. At any rate, what we 
hope will happen is that those with* psychometric skills and those 
who$e expertise is in the areas of learning and instruction will meet 
intellectually and jointly continue the task of focusing educational 
productivity on learning tasks. 
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